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Abstract 

Accurate  modeling  of  the  auroral  region  is  an  important  component  to  mitigate 
risk  to  space-based  assets  and  communications  systems.  Many  auroral  models  have  been 
developed,  but  little  effort  has  been  made  to  validate  their  accuracy.  The  location  of  the 
equatorward  boundary  of  the  auroral  oval  provides  an  appropriate  means  to  measure  the 
accuracy  of  auroral  models. 

In  this  study,  the  equatorward  boundary  was  represented  by  the  location  at  which 
the  energy  flux  measured  by  DMSP  satellites  exceeded  a  fixed  threshold  of  0.4  erg/cm'/s. 
Magnetic  latitude  (MLAT)  coordinates  were  obtained  from  more  than  4,000  orbits 
through  the  polar  region  of  the  Northern  Hemisphere  and  compared  to  the  outputs  of  five 
auroral  precipitation  models.  The  models  tested  in  this  study  included:  the  original  Hardy 
auroral  model  of  1985  (OH),  the  OVATION  Prime  model  (OP),  the  2008  adaptation  to 
the  Hardy  model  (NH),  the  Space  Weather  Modeling  Framework  Ring-Current  model 
(SWMF),  and  the  Assimilated  Mapping  of  Ionospheric  Electrodynamics  model  (AMIE). 

Each  model’s  energy  flux  output  was  compared  to  DMSP  data  in  a  variety  of 
categories  including:  high,  moderate,  and  low  Kp  index  conditions;  dawn,  day,  and  dusk 
magnetic  local  time  (MLT)  sectors;  and  all  four  seasons.  Differences  between  the  model 
and  satellite  data  were  quantified  and  catalogued.  A  prediction  efficiency  (PE)  score  and 
was  also  calculated  for  each  model  and  used  to  assess  relative  accuracy  in  the  various 

categories.  The  statistical  results  using  0.4  erg/cm'/s  were  validated  by  repeating  the 
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analysis  using  0.6  erg/cm'/s  as  the  energy  flux  threshold. 
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In  each  of  the  19  Kp-MLT-season  categories  combinations  considered  in  this 
study,  no  singular  model  garnered  a  preponderance  of  the  highest  PE  scores,  when  using 
the  lower  energy  flux  threshold.  The  SWMF  model  received  the  highest  PE  score  in  4 
categories,  the  NH  model  in  4  categories,  the  OH  model  in  5  categories,  and  the  OP 
model  in  6  categories.  When  the  higher  flux  threshold  was  used,  the  OP  model  received 
the  majority  of  the  highest  PE  scores. 

The  model  with  the  highest  overall  prediction  efficiency  score  was  the  OP  model 
(0.55).  The  OH  model  (0.5 1)  had  the  second  highest  score,  and  it  was  followed  by  the 
NH  model  (0.45).  The  OP  model’s  performance  was  well  corroborated  using  the 
0.6  erg/cm  /s  threshold.  Thus,  the  OP  model  is  therefore  deemed  to  be  the  most  accurate 
of  the  models  tested  in  this  study  and  is  most  suitable  for  operational  auroral  forecasting. 
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COMPARATIVE  STATISTICAL  ANALYSIS  OF  AURORAL  MODELS 


I.  Introduction 

1.1  Motivation 

Several  governmental  agencies,  including  the  Department  of  Defense  (DoD), 
increasingly  rely  on  highly  technological  assets  operating  in  the  space  environment  to 
achieve  national  security  objectives,  and  satellite  operations  have  become  an  integral 
component  of  the  execution  of  the  national  defense  strategy  of  the  United  States.  An 
increased  dependence  on  these  assets  requires  that  efforts  to  mitigate  risk  remain  a  top 
priority. 

Some  of  space  asset’s  vulnerability  is  related  to  the  highly  variable  conditions  of 
the  space  environment,  often  termed  space  weather.  Space  weather,  broadly  defined, 
refers  to  fluctuations  in  solar  activity  and  solar  wind  conditions  that  couple  to  earth’s 
atmosphere  via  magnetospheric  and  ionospheric  processes.  This  coupling  can  cause 
substantial  increases  in  the  density  of  highly  energetic,  charged  particles  in  the  satellite 
environment.  In  extreme  cases,  the  conditions  can  become  hazardous  enough  to  damage 
to  the  fragile  electronics  on  which  the  asset  relies,  jeopardizing  its  sustainability. 

A  complete  theoretical  understanding  of  the  interplay  among  the  complex 
physical  processes  that  influence  space  weather  conditions  is  not  imminent.  Thus,  many 
research  efforts  have  focused  on  developing  and  honing  various  models  to  better  predict 
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and  forecast  the  onset,  duration,  and  severity  of  changes  in  the  earth’s  space  weather 
environment.  The  DoD  has  assumed  a  significant  role  in  the  development  of  better 
models  because  the  ability  to  effectively  and  reliably  model  space  weather  phenomena 
provides  the  framework  from  which  a  strategy  to  protect  space  assets  can  be  formulated. 
From  a  sustainability  perspective,  an  accurate  prediction  of  unfavorable  conditions  can 
afford  satellite  operators  the  time  to  execute  contingency  processes  to  shut  down  and/or 
shield  the  assets.  But  accurate  modeling  can  also  enhance  operations  by  predicting 
periods  where  an  asset’s  performance  (or  an  adversary’s  asset)  will  be  degraded. 

1.2  Research  Topic 

Auroral  precipitation  models  have  been  extensively  used  for  estimating  GPS  and 
other  communication  satellite  disturbances  (Newell  et  ah,  2010a).  The  auroral  oval 
represents  one  candidate  within  the  space  weather  framework  well  suited  for  modeling 
because  its  characteristics  are  a  direct  manifestation  of  the  space  weather  conditions.  In 
fact,  auroral  precipitation  models  have  existed  for  several  decades,  and  studies  of  the 
statistical  systematics  pertaining  to  high-latitude  particle  precipitation  have  proved  very 
useful  to  the  space  weather  community  (Hardy  et  ah,  2008).  The  oval  in  question  refers 
to  an  annular  region  of  significant  thickness  that  encircles  the  magnetic  north  and  south 
poles.  Here,  charged  particles,  originating  in  the  earth’s  magnetosphere,  constantly 
precipitate  into  the  upper  atmosphere  along  the  planet’s  geomagnetic  field  lines. 

This  study  specifically  focuses  on  the  systematic  behavior  of  diffuse  auroral 
precipitation,  which  is  a  pervasive  and  large-scale  phenomenon.  Even  during  quiet 
conditions,  energy  flux  measurements  can  be  used  to  discern  the  location  of  the 
precipitation  boundaries.  However,  during  geomagnetic  storming,  the  equatorward 
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boundary  of  the  oval  predictably  expands,  resulting  in  a  physical  manifestation  that  is 
well  suited  for  study  and  analysis. 

The  scope  of  this  study  supports  taking  a  limited  approach  with  regard  to  the 
complex  characteristics  of  the  particles  precipitating  into  equatorward  region  of  the  oval. 
Integral  energy  fluxes  will  be  used  based  upon  the  commonly  accepted  assumption  of 
isotropic  particle  precipitation.  Furthermore,  no  distinction  will  be  made  among  the 
various  types  and  characteristics  of  the  particles  forming  the  aurora.  To  that  end,  only 
total  particle  counts  will  be  analyzed,  and  ion  and  electron  energy  fluxes  will  be  summed 
together.  Thus,  the  research  conducted  here  will  purposely  not  showcase  the  full 
capability  of  many  of  the  models  under  review,  some  of  which  take  great  effort  to  model 
the  behavior  of  specific  particle  types.  Nevertheless,  taking  this  limited  view  is  justified 
because  there  have  been  few  objective  studies  designed  to  measure,  even  in  the  most 
generic  sense,  how  accurately  models  depict  the  extent  of  the  oval. 

1.3  Research  Objective 

This  study  seeks  to  objectively  quantify  the  accuracy  of  different  auroral  models. 
Researchers,  over  the  years,  have  developed  a  multitude  of  different  auroral  precipitation 
models  based  on  a  variety  of  different  methodologies.  Generally  speaking,  each  model 
possesses  its  own  strengths  and  weaknesses  by  virtue  of  its  design.  However,  only 
recently  have  efforts  been  made  to  compare  how  well  different  models  predict  changes  in 
the  auroral  oval  in  response  to  various  geomagnetic  conditions. 

In  2010,  Newell  et  al.  conducted  a  comparative  study  of  four  auroral  precipitation 
models.  This  research  dealt  with  how  accurately  each  model  predicted  instantaneous 
auroral  power  by  comparing  each  model’s  results  to  synoptic  UV  images  from  NASA’s 
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Polar  satellite.  In  this  study,  a  similar  methodology  will  be  applied  to  a  different  dataset. 
Integral  DMSP  energy  flux  data  will  be  used  to  identify  the  spatial  extent  of  the 
equatorward  boundary  at  a  specific  point  along  the  oval’s  boundary.  Although  a  single 
DMSP  satellite’s  transit  through  the  polar  region  is  a  poor  synoptic  representation  of  the 
oval,  this  specific  crossing  point  can  be  easily  compared  to  the  corresponding  output  of  a 
precipitation  model’s  output,  allowing  a  suitable  comparison  between  the  two.  The 
DMSP  data  archives  contain  a  large  amount  of  easily  accessible  data,  which  will  permit 
the  creation  of  a  sizeable  and  statistically  significant  dataset  that  includes  a  variety  of 
conditions  obtained  at  a  variety  of  locations  around  the  polar  oval. 

The  equatorward  extent  of  the  auroral  oval  will  be  detennined  using  the  DMSP 
satellites’  energy  flux  data.  The  DMSP  results  will  be  compared  to  the  energy  flux 
output  of  five  auroral  precipitation  models  given  the  same  input  parameters.  The  five 
models  are:  (1)  the  Hardy  Kp  model  from  1985;  (2)  a  2008  update  to  the  Hardy  Kp 
model,  provided  by  Dr.  Chin  at  the  Air  Force  Research  Laboratory  (AFRL);  (3)  the 
OVATION  Prime  model,  developed  by  Newell  and  colleagues;  (4)  a  physics-based 
model  developed  by  coupling  data  from  the  Space  Weather  Modeling  Framework 
(SWMF)  to  a  Ring  Current  model  maintained  at  the  Community  Coordinated  Modeling 
Center  (CCMC)  at  NASA-Goddard;  and  (5)  results  obtained  from  the  Assimilative 
Mapping  of  Ionospheric  Electrodynamics  (AMIE)  program  provided  by  colleagues  at  the 
University  of  Michigan.  Quantitative  analysis  of  difference  between  the  models’ 
boundary  locations  compared  to  the  determined  boundary  using  the  DMSP  flux  data  will 
pennit  an  objective,  albeit  partial,  assessment  of  each  model’s  accuracy. 
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1.4  Document  Structure 


This  document  is  arranged  into  five  chapters.  Chapter  II  includes  the  theoretical 
framework  of  the  physical  processes  that  create  the  auroral  oval.  It  additionally  discusses 
other  important  concepts  associated  with  auroral  modeling  and  provides  background 
information  on  each  of  the  models.  Chapter  III  explains  the  methods  used  to  obtain  and 
analyze  the  data  for  this  study.  Chapter  IV  contains  the  results  of  the  analysis  and  their 
relevance  and  includes  a  determination  of  the  most  accurate  model.  In  Chapter  V,  a 
concise  summary  of  the  results  and  suggestions  for  subsequent  experimentation  and 
investigation  will  be  provided. 
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II.  Theory  and  Background 


This  chapter  provides  theoretical  material  and  background  information  relevant  to 
this  study.  It  is  divided  into  four  main  sections.  The  first  section  contains  a  description 
of  important  auroral  processes,  including  the  solar  wind,  magnetic  reconnection, 
magnetospheric  particle  flow,  coupling  to  the  ionosphere,  and  auroral  currents  and 
emissions.  The  second  section  of  this  chapter  introduces  specific  concepts  associated 
with  the  auroral  oval’s  boundary  and  discusses  a  useful  taxonomy.  In  the  third  section, 
background  information  relevant  to  auroral  observation  and  measurement  is  provided. 
This  includes  describing  the  Kp  index  and  DMSP  data  characteristics.  The  final  section 
provides  background  infonnation  on  each  of  the  five  auroral  models  that  will  be  utilized 
in  this  study. 

2.1  Theoretical  Background 

From  a  purely  phenomenological  point  of  view,  the  aurora  is  simply  the  emission 
of  energy  resulting  from  atomic  and  molecular  excitation  at  altitudes  between  100  and 
300  km  (Prolss,  2004).  The  neutral  gas  excitation  occurs  when  energetic  charged 
particles  precipitating  into  the  earth’s  atmosphere  collide  with  the  various  atmospheric 
gases  (Prolss,  2004).  The  aurora’s  most  noticeable  characteristics  are  the  vibrant  bands 
of  light  arcing  throughout  the  polar  night  sky.  But,  this  highly  dynamic  an  often  times 
finely  structured  aurora  represents  a  specific  type  of  aurora — discrete  aurora. 

In  the  late  1960s,  the  existence  of  a  relatively  structureless  band  of  large-scale 
auroral  precipitation  was  observed  with  both  ground-  and  space-based  sensors.  The  lack 
of  structure  led  to  its  eventual  classification  as  diffuse  aurora.  Shortly  thereafter,  a 
satellite  launched  as  part  of  the  International  Satellites  for  Ionospheric  Studies  (ISIS) 
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program  observed  a  global  pattern  of  structureless,  auroral  luminosity  encircling  the 
entire  geomagnetic  pole.  This  confirmed  the  diffuse  auroral  phenomena  was  occurring  in 
both  the  day  and  night  hemispheres  (Paschmann,  2003).  Today,  through  comprehensive 
imaging  studies,  it  is  known  the  auroral  oval  describes  of  an  elliptical  ring  of  varying 
thickness  whose  center  is  displaced  anti-sunward  of  the  magnetic  pole.  The  radii  of  the 
ring’s  inner  and  outer  boundaries  expand  and  contract  in  response  to  changing  solar  wind 
and  magnetospheric  parameters.  The  thickest  part  of  the  ring  and  its  most  equatorward 
extent  are  always  found  on  the  antisunward  side  of  earth.  The  schematic  in  Figure  1 
depicts  the  general  shape  of  the  polar  oval,  showing  how  it  varies  diurnally. 


Figure  1.  Schematic  of  polar  oval  showing  the  characteristic  diurnal  migration  (Whalen,  2001). 

The  particles  responsible  for  the  diffuse  aurora  are  prevalent  in  the  equatorward 
regions  of  the  oval  and  precipitate,  to  some  extent,  during  all  levels  of  geomagnetic 
conditions.  Even  though  the  diffuse  emissions  are  typically  too  dim  to  observe  with  the 
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naked  eye,  measurements  have  since  determined  the  diffuse  aurora  is  responsible  for 
50-80%  of  the  total  energy  input  into  the  upper  atmosphere  (Whalen,  2001).  Thus,  the 
equatorward  boundary  of  the  oval  is  one  of  the  most  well-suited  locations  to  measure  the 
performance  of  various  auroral  models  during  a  variety  of  conditions. 

The  molecular  light  emissions  observed  on  the  earth,  whether  part  of  the  diffuse 
or  discrete  aurora,  are  actually  the  last  link  in  a  long  chain  of  processes.  The  schematic 
shown  in  Figure  2  describes  this  linkage.  An  understanding  of  how  the  equatorward 
boundary  of  the  oval  changes  requires  knowledge  of  the  solar  wind  and  the  Interplanetary 
Magnetic  Field  (IMF),  the  individual  and  coupled  dynamics  of  the  magnetosphere  and 
ionosphere,  and  the  complicated  chemical  processes  occurring  in  the  earth’s  atmosphere. 
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Figure  2.  Auroral  Process  Chain  (Akasofu,  1981). 


2.1.1  The  Solar  Wind 

The  auroral  process  begins  when  the  continual  stream  of  charged  particles  in  the 
solar  wind  interacts  with  the  earth’s  magnetic  field.  The  solar  wind  is  highly  conductive, 
collisionless,  magnetized  plasma  emitted  from  the  sun’s  corona.  Because  the  plasma  is 
highly  conductive,  the  sun’s  magnetic  field  is  carried  along  by  it.  The  sun’s  rotation 
causes  the  magnetic  field  lines  to  bend  into  the  spiral  shape  depicted  in  Figure  3.  This 
accounts  for  the  characteristic  shape  of  the  Interplanetary  Magnetic  Field  (IMF),  which 
penneates  the  entire  solar  system.  The  specific  orientation  of  the  IMF  with  reference  to 
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the  ecliptic  plane  (i.e.,  its  /^--component)  is  associated  with  geomagnetic  conditions 
favorable  to  producing  aurora.  Additional  information  can  be  found  in  Tascione  (1998). 


Solar  Wind 


Figure  3.  Depiction  of  IMF’s  spiral  orientation  (Tascione,  1988). 

The  vast  majority  of  the  solar  wind  is  deflected  around  the  earth.  The  impinging 
pressure  of  the  solar  wind  particles  exerts  a  force  on  the  earth’s  dipolar  field,  forming  a 
bow  shock,  analogous  to  the  aerodynamic  shock  fonned  around  a  blunt  obstacle 
(Tascione,  1988).  Inside  of  this  shock  boundary,  there  exists  a  finite  region  where  the 
largely  dynamic  pressure  of  the  solar  wind  equals  the  oppositely  directed  magnetic  field 
pressure  of  the  earth.  This  results  in  the  formation  of  the  magnetopause,  a  distinct 
boundary  that  shields  the  magnetosphere  from  the  vast  majority  of  solar  wind  particles. 
The  sunward  side  of  the  magnetosphere  is  compressed  into  an  ellipsoid  shape.  On  the 
anti-sunward  side,  the  magnetosphere  stretches  well  beyond  60  RE,  forming  a  long 
magnetotail  in  which  the  geomagnetic  field  lines  are  extremely  elongated  (Prolss,  2004). 
The  magnetospheric  cusp  describes  the  separation  region  between  the  field  lines 
comprising  the  day-  and  night-sides  of  the  magnetosphere.  A  schematic  representing  the 
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compression  of  field  lines  on  the  sunward  side  and  extension  of  field  lines  on  the  anti- 
sunward  side  is  provided  in  Figure  4. 


Figure  4.  Depiction  of  earth's  magnetosphere  (Prolss,  2004). 


2.1.2  Magnetic  Reconnection 

Due  to  the  magnetopause,  the  solar  wind  plasma  cannot  precipitate  directly  into 
the  auroral  oval,  but  it  is  the  only  source  of  plasma  that  can  replenish  particle  losses 
occurring  in  the  continuous  auroral  process  (Prolss,  2004).  This  condition  necessitates 
the  existence  of  a  coupling  mechanism  that  allows  some  particles  to  cross  the 
magnetopause  and  to  enter  the  earth’s  magnetosphere.  Research  has  shown  the  rate  of 
inelastic  losses  in  the  ionosphere  (e.g.,  auroral  emissions)  requires  approximately  0.1%  of 
the  incident  solar  wind  particles  to  sustain  the  process  (Tascione,  1988). 

It  is  largely  believed  that  an  opening  and  reconnection  process  overcomes  the 
otherwise  solid  boundary  of  the  magnetopause  and  supplies  the  magnetosphere  with 
approximately  90%  of  its  energy  (Prolss,  2004).  The  earth’s  magnetic  field  can  be 
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thought  of  as  a  large  magnetic  dipole  with  an  asymmetric,  albeit  closed,  magnetic  field. 
Both  footpoints  of  a  closed  field  line  tenninate  on  the  earth’s  surface.  However,  the 
theoretical  concept  of  magnetic  opening  provides  a  mechanism  to  defeat  the  otherwise 
closed  system. 

At  places  where  the  IMF  and  the  earth’s  geomagnetic  field  have  opposite  polarity, 
a  neutral  point  can  form  that  breaks  apart  a  closed  field  line.  This  allows  the  newly  freed 
ends  to  be  swept  along  with  the  solar  wind  stream  toward  the  magnetotail.  Solar  wind 
plasma  becomes  associated  with  this  open  field  line,  and  when  the  broken  field  line 
reconnects  somewhere  on  the  anti-sunward  side  of  the  earth,  the  plasma  is  introduced  into 
the  magnetosphere.  This  process,  tenned  magnetic  reconnection,  replenishes  the 
reservoir  of  particles  and  also  releases  energy  (Prolss,  2004).  Reconnection  occurs  within 
200  Re,  but  near  earth  reconnection  (-20-30  Re)  often  correlates  to  faster  convective 
flows  within  the  plasma  sheet  and  an  expansion  of  the  auroral  oval  (Ohtani  et  al.,  2004). 

Neutral  points  may  form  during  both  IMF-northward  and  IMF-southward 
conditions.  However,  it  has  been  demonstrated  empirically  that  when  the  earth  enters  a 
region  of  southward  orientation,  more  particles  enter  the  magnetosphere,  and 
geomagnetic  stonning  is  intensified.  Additional  information  regarding  reconnection  is 
contained  in  Prolss  (2004)  and  Paschmann  (2003). 

2.1.3  Magnetospheric  Particle  Flow 

Reference  was  made  in  Section  2.1.1  to  the  stretching  of  the  magnetic  field  lines 
on  the  anti-sunward  side  of  earth.  The  highly  extended  lines  in  the  magnetotail  are 
characterized  by  a  sharp  reversal  in  the  tangential  component  of  their  polarity.  The 
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graphic  shown  in  Figure  5  demonstrates  the  extreme  stretching  of  the  field  lines  in  the 
magneto  tail. 


Re 


Figure  5.  Geomagnetic  field  lines  in  the  x-z  plane  of  the  earth’s  magnetotail  (Prolss,  2004). 

This  feature  creates  a  region  along  the  ecliptic  plane  known  as  the  magnetically 
neutral  sheet,  which  pennits  the  formation  of  a  surface  current  that  flows  across  the  tail 
from  dawn  to  dusk.  The  current  creates  a  convection  electric  field  oriented  in  the  same 
direction.  The  perpendicular  orientation  of  the  electric  (east-to-west)  and  magnetic 
(south-to-north)  fields  in  and  near  the  ecliptic  plane  of  the  magnetotail  results  in  both  ions 
and  electrons  in  this  region  drifting  toward  the  earth  as  a  result  of  an  Ex  B  drift.  This 
drift  is  given  by 

E  x  B 

UEXB  =  B2  (1) 

in  which  there  is  no  dependence  upon  particle  charge  or  energy. 

As  the  particles  drift  closer  to  earth,  two  things  occur.  First,  they  are  heated  and 
compressed  but  maintain  a  nearly  Maxwellian  energy  distribution  (Paschmann,  2003). 
Second,  they  encounter  a  region,  as  is  shown  in  Figure  5,  where  the  dipole  field  is  far  less 
elongated  than  it  was  in  far  regions  of  the  magnetotail.  In  this  region,  there  exists  a 
stronger  magnetic  field  gradient  perpendicular  to  the  field  lines.  The  increase  in  energy 
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and  the  changing  shape  of  the  magnetic  field  cause  the  gradient  and  curvature  drift  to 
become  dominant.  This  quantity  is  often  expressed  as  total  particle  drift  and  is  given  by 


Ud  =  \2^J  (v'+2vll)— g3—  (2) 

which  is  dependent  upon  both  particle  charge  and  kinetic  energy.  For  the  purposes  of  the 
discussion  salient  to  this  study,  it  is  sufficient  to  introduce  a  simplifying  assumption,  as  is 
done  by  Prolss  (2004).  The  total  drift  in  the  vicinity  of  the  mirror  points  is  significantly 
less  than  that  found  at  the  equatorial  plane  due  to  the  parallel  component  of  the  velocity 
approaching  zero  and  the  reduction  in  the  field  gradient.  To  reasonable  approximation, 
then,  it  is  possible  to  ignore  curvature  drift  component  in  the  Equation  (2),  in  which  case, 
the  total  drift  is  equal  to  the  gradient  drift.  Between  approximately  10-12  Re,  the 
dominance  of  the  gradient  drift  begins  to  manifest  itself  on  the  highest  energy  particles. 
They  are  turned  perpendicularly  to  the  orientation  of  the  convective  flow,  and  because  of 
the  charge  dependence,  positively  charged  ions  turn  to  the  west  and  electrons  turn  toward 
the  east  (Paschmann,  2003). 

The  dominance  of  the  gradient  drift  over  the  ExB  drift  as  a  function  of  particle 
energy  ultimately  implies  a  point  of  closest  approach  for  the  lowest  energy  ions  and 
electrons.  This  region  in  the  nightside  magnetosphere  maps  along  the  associated  field 
lines  and  demarcates  the  equatorward  boundary  of  diffuse  precipitation  in  the  oval 
(Prolss,  2004).  The  following  equation  establishes  the  distance  from  earth  at  which  this 
point  occurs: 

3  Wverv 

L  =  W tt,  (3) 
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where  Wperp  is  the  kinetic  energy  perpendicular  to  the  magnetic  field  and  Econ  is  the 
convective  electric  field  mapped  from  the  solar  wind.  This  relation  clearly  demonstrates 
two  important  characteristics.  First,  as  the  convective  electric  field  increases,  as  it  does 
during  periods  of  increased  geomagnetic  activity,  the  distance  of  closest  approach 
decreases.  This  gives  particles  of  certain  energy  access  to  field  lines  whose  footpoints 
map  to  lower  latitudes,  and  the  equatorward  boundary  of  the  auroral  oval  expands 
(Prolss,  2004). 

Additionally,  the  effect  of  the  energy  tenn,  as  was  discussed  earlier,  implies  lower 
energy  particles  can  be  found  closer  to  the  earth.  This  has  been  validated  through 
observation.  The  graphic  in  Figure  6  demonstrates  electron  flux  counts  in  discrete  energy 
channels.  The  arrows  show  the  onset  of  precipitating  electrons.  The  precipitation  onset 
of  more  energetic  electrons  is  observed  at  more  poleward  latitudes,  which  is  consistent 
with  their  association  with  field  lines  farther  from  the  earth.  This  has  been  found  to  be  a 
energy  onset  pattern  that  is  particularly  characteristic  of  the  equatorward  boundary 
located  in  the  evening  and  night  sectors  (Gussenhoven  et  ah,  1981). 

2.1.4  Ionospheric  Coupling 

Charged  particles  spiraling  along  field  lines  will  maintain  their  first  adiabatic 
invariant  under  static  conditions.  This  means  that  as  the  intensity  of  the  magnetic  field 
changes,  the  particle’s  perpendicular  velocity  increases.  Conservation  of  energy  requires 
the  parallel  component  of  the  spiraling  particle’s  velocity  to  eventually  go  to  zero,  and  an 
oppositely  directed  gradient  force  reflects  the  particle.  This  condition,  by  itself,  would 
result  in  a  rapid  emptying  of  the  loss  cone  and  an  unsustainable  auroral  process.  In 
effect,  most  particles  would  bounce  between  their  mirror  points  well  above  the 
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atmospheric  regions  where  the  aurora  forms.  Since  the  auroral  is  pervasive,  there  must 
exist  some  mechanism  to  refill  the  loss  cones.  Scattering  processes  are  thought  to 
provide  the  means  by  which  the  loss  cone  population  is  replenished,  sustaining  the 
auroral  process. 
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Figure  6.  DMSP  satellite  particle  counts  in  a  few  of  the  sensor’s  discrete  energy  bins  for  a  satellite 
pass  in  the  night  sector.  Arrows  correspond  to  coordinates  at  which  significant  increase  in  number 
flux  occurs.  Boundary  associated  with  higher  energy  particles  is  observed  at  higher  latitudes 
consistent  with  curvature  drift  dominance  (Gussenhoven  et  al.,  1981). 


For  the  more  massive  ions,  scattering  largely  occurs  because  of  the  specific  shape 
of  the  field  lines  in  the  magnetotail.  Although  they  are  closed,  the  field  lines  in  question 


15 


exhibit  a  sharp  reversal  in  the  field  lines  of  the  nightside  magnetosphere  that  can  result  in 
pitch  angle  scattering  into  the  loss  cone.  This  occurs  because  the  typical  ion  gyroradius, 
which  is  much  larger  than  that  of  an  electron,  becomes  comparable  to  the  radius  of 
curvature  of  the  magnetic  field  in  the  narrow  region  near  the  ecliptic  plane  (Paschmann, 
2003). 


As  is  given  by  Prolss  (2004),  particle  motion  can  be  described  by  a  ratio  of  the 
field  line  curvature  to  the  particle  gyroradius: 

qLBl 


K2  = 


(4) 


m  v  Blobe 

where  Bn  represents  the  perpendicular  magnetic  field  strength  in  the  region  spanning  the 
current  sheet  and  Blobe  is  the  larger  tangential  field  strength  immediately  above  and 
below  the  current  sheet.  L  represents  the  scale  length  (~  1 000- 1 0.000  km)  for  the  reversal 
of  polarity  across  the  current  sheet.  When  k2  S  1,  strong  pitch-angle  scattering  occurs. 
For  protons  in  the  mid-tail  regions  of  the  ionosphere,  this  relation  is  satisfied  for  a  wide 
variety  of  energies  spanning  30  eV  to  190  keV,  which  agree  closely  with  observed  proton 
energies  in  the  equatorward  region  of  the  auroral  oval  (Paschmann,  2003). 

For  electrons  in  the  plasma  sheet,  the  mechanism  that  fills  the  loss  cone  for  the 
diffuse  aurora  is  thought  to  be  pitch  angle  diffusion  brought  about  by  wave-particle 
scattering.  Scattered  electrons  may  obtain  velocities  that  are  parallel  to  the  magnetic  field 
lines  in  the  region.  This  change  in  velocity  puts  the  electrons  into  the  loss  cone,  and  they 
are  allowed  to  precipitate  down  into  the  upper  atmosphere  (Prolss,  2004).  This  process  is 
shown  schematically  in  Figure  7. 
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Pitch-angle  scattering,  therefore,  forms  the  critical  bridge  between  the  plasma 
sheet  along  the  equatorial  plane  and  the  upper  atmosphere.  Additionally,  the  following 
conclusions  can  be  drawn.  The  distant  regions  of  the  magnetosphere  contribute  small 
electron  energy  fluxes  to  the  ionosphere.  Precipitation  increases  as  a  function  of 
increased  thermal  energy  as  the  particles  are  brought  earthward,  where  perturbations  in 
parallel  velocities  are  sufficient  to  move  the  particles  into  their  respective  loss  cones 
(Paschmann,  2003).  The  loss  cone  is  a  function  of  the  ratio  between  the  magnetic  fields 
strength  of  a  field  line  at  the  equator  to  the  field  strength  of  the  same  field  line  mapped 
into  the  polar  ionosphere  using  a  quasi-dipole  approximation.  Typical  field  strength 
ratios  are  on  the  order  of  10'4  resulting  in  loss  cone  angles  ~  1°  (Paschmann,  2003).  For 
an  isotropic  temperature  distribution  of  electrons,  the  fraction  of  particles  in  the  loss  cone 
is  on  the  order  of  lxl O'4. 

2.1.5  Plasma  Sheet  Topology 

As  the  primary  source  for  diffuse  auroral  particles,  the  mapping  of  plasma  sheet  to 
the  polar  ionosphere  is  important.  Crude  approximations  can  be  made  using  a  pure 
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dipole  field.  The  following  equation  expresses  the  magnetic  latitude  of  a  footpoint  to  the 
corresponding  L-shell  (multiple  of  Re)  at  a  height  h  above  the  earth’s  surface 

[re+J l 

f  =  arccos  J'kTT  <5) 

Using  this  approximation,  the  inner  and  middle  portions  of  the  plasma  sheet,  which  exist 
between  approximately  5  and  40  RE,  are  threaded  by  closed  geomagnetic  field  lines 
which  correspond  to  geomagnetic  latitudes  (MLATs)  between  approximately  63-80°.  In 
order  to  account  for  the  deviations  from  a  pure  dipole,  various  models  have  been 
developed.  Figure  8  depicts  field  lines  as  a  function  of  MLAT.  As  this  model 
demonstrates,  using  a  simple  dipole  approximation  introduces  errors  at  MLATs  greater 
than  70°. 


Figure  8.  Geomagnetic  field  model  accounting  for  asymmetry  during  quiet  conditions  (Kp  0). 
Latitudes  depicted  are  invariant  (adapted  from  Hargreaves,  1995). 


Under  all  but  the  most  extreme  conditions,  the  auroral  zone  exists  between 
60-80°  MLAT.  As  was  discussed  in  the  previous  section,  its  equatorward  boundary 
corresponds  to  the  most  earthward  region  of  precipitating  particles,  which  conventionally 
describes  the  boundary  region  of  the  plasma  sheet.  The  connection  between  the 
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characteristics  of  the  particles  in  the  plasma  sheet  and  the  diffuse  auroral  oval  was  made 
in  the  mid-1970s  (Newell  et  ah,  1996).  Since  then,  subsequent  studies  have  confirmed 
the  measured  electron  energies  in  the  diffuse  aurora  are  characteristic  of  the  plasma  sheet 
population. 

2.1.6  Auroral  Current  System  and  Emissions 

The  regions  of  precipitating  particles  in  the  auroral  zone  fonn  the  current  linkage 
between  the  ionosphere  and  magnetosphere.  The  energy  transfer  between  the 
magnetosphere  and  the  ionosphere  is  accomplished  via  field-aligned  currents  (FAC). 
Figure  9  shows  a  generic  schematic  of  the  FAC  flowing  into  and  out  of  the  ionosphere  in 
the  polar  region. 


Figure  9.  Characteristic  polar  region  current  flows.  Latitude  is  invariant,  (adapted  from  Schunk, 
2009). 


The  FAC  are  an  integral  part  of  a  complex  current  system  that  involves  the  entire 
polar  cap.  They  are  divided  into  two  regions.  Region  1  currents  are  everywhere 
poleward  of  Region  2  currents.  As  depicted  in  Figure  9,  on  the  dawn  side  of  the  polar 
cap,  current  flows  into  the  ionosphere  in  Region  1  and  out  of  the  ionosphere  in  Region  2. 
This  system  is  reversed  on  the  dusk  side.  The  main  distinction  between  the  regions  is 
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that  Region  1  currents  couple  to  the  solar  wind  through  open  magnetic  field  lines, 
whereas  Region  2  field  lines  are  entirely  part  of  the  closed  system  of  the  inner 
magnetosphere  (Paschmann,  2003). 

The  FAC  are  primarily  carried  by  electrons.  When  current  is  flowing  into  the 
ionosphere,  the  upward  flowing  electrons,  sourced  in  the  ionosphere,  are  comparatively 
cold  and  responsible  for  little  neutral  gas  excitation.  More  energetic  electron  populations 
from  the  magnetosphere  precipitate  in  regions  where  current  is  flowing  out  of  the  polar 
cap.  Studies  have  confirmed  that  there  is  very  little  field-aligned  acceleration  existing  in 
the  quasi-dipolar  region  of  the  inner  magnetosphere  (Paschmann,  2003).  Thus,  under 
quiescent  conditions,  thennalized  electrons  carry  the  current,  and  diffuse  aurora  is 
pervasive.  However,  during  active  periods,  the  convective  flow  from  the  plasma  sheet  is 
increased,  and  current  continuity  must  be  maintained  through  the  formation  of  magnetic 
field-aligned  electric  fields.  These  are  the  electric  fields  that  result  in  accelerated 
particles  responsible  for  the  visible,  dynamic  auroral  displays  (Paschmann,  2003). 

In  order  to  maintain  charge  continuity,  inbound  and  outbound  currents  are 
connected  with  horizontal  currents  flowing  across  the  geomagnetic  field  lines  in  the  polar 
ionosphere  (Paschmann,  2003).  The  Hall  and  Pedersen  currents  complete  the  auroral 
circuitry  linking  the  magnetosphere  and  ionosphere.  Additional  discussion  of  these 
current  systems  can  be  found  in  Schunk  (2009). 

An  in-depth  review  of  the  ionospheric  chemistry  that  results  in  the  auroral 
radiation  exceeds  the  scope  of  this  research  and  will  not  be  included  here.  These 
processes  are  discussed  at  great  length  in  Schunk  (2009).  One  point  will  be  clarified. 
Auroral  emissions  occur  between  -90-200  km,  approximately  700  km  below  the  DMSP 
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satellite  orbits.  As  can  be  confirmed  by  Equation  (4),  when  considering  altitudes  much 
less  than  one  earth  radius,  the  field  lines  are  virtually  perpendicular  to  the  earth’s  surface. 
Thus,  the  location  of  a  satellite  in  low-earth  orbit  correlates  closely  to  the  region  of  the 
ionosphere  where  the  auroral  energy  is  radiated. 

2.2  Auroral  Oval  Boundaries 

By  the  mid-1980s,  numerous  studies  had  confirmed  that  the  entire  auroral  oval, 
including  the  subvisual  diffuse  aurora  found  near  the  equatorward  boundary,  migrated 
north-  and  southward  in  response  to  magnetic  substorming  (Newell,  et  al.  1996). 
Replicating  the  full  extent  of  the  boundary’s  shifting  nature  has  remained  a  primary  focus 
of  early  and  more  recent  modeling  efforts. 

Gussenhoven  et  al.  (1981)  were  responsible  for  one  of  the  first  research  efforts  of 
this  sort.  Gussenhoven’s  study  sought  to  categorize  the  oval’s  boundaries  using  data 
gained  from  DMSP/F2  satellite  passes.  The  boundary  coordinates  were  assigned  to  the 
location  where  the  number  flux  exceeded  10  elec/cnr/s/sr,  a  value  chosen  to  represent  a 
significant  rise  above  the  background  flux  counts  (Gussenhoven,  1981).  The  results  of 
the  research  confirmed  the  existence  of  a  strong  correlation  between  the  equatorward 
boundary’s  location  and  the  Kp  index.  Incidentally,  this  study  also  discovered  the 
boundary  in  the  pre-noon  MLT  sectors  was  more  difficult  to  detennine  than  in  the 
evening  sectors,  an  issue  which  will  be  discussed  in  a  later  section. 

In  1996,  Newell  and  colleagues  conducted  an  extensive  study  aimed  at  objectively 
categorizing  the  various  boundaries  that  had  been  discovered  within  the  nightside  auroral 
oval.  Physical  manifestations  throughout  the  inner  magnetosphere  associate  to  specific 
qualities  observed  in  the  energy  signatures  of  the  precipitating  particles.  Newell’s  study 
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associates  various  theoretical  precepts  with  observed  features  found  at  locations  within 
the  auroral  oval.  Conveniently,  Newell’s  boundaries  were  fonnulated  based  upon  energy 
flux  measurements,  consistent  with  the  data  obtained  in  this  study  and,  thus,  provide  a 
useful  framework. 

One  example  of  the  type  of  categorization  conducted  in  Newell’s  study  pertains  to 
the  point  where  the  reduced  curvature  of  the  magnetic  field  lines  no  longer  causes  pitch 
angle  scattering  of  the  ions  near  the  neutral  sheet.  This  point  maps  to  an  observable  ion 
boundary  in  the  auroral  oval,  known  as  the  ion  isotropy  boundary.  Newell’s  study  also 
defined  two  boundaries  relating  to  the  oval’s  equatorward  regions.  The  most 
equatorward  boundary  corresponds  to  the  convection  boundary  formed  between  the  zero- 
energy  population  of  the  plasmasphere  and  the  lowest  energy  particles  of  the  plasma 
sheet. 

As  was  described  in  an  earlier  section,  low  energy  particles  are  less  influenced  by 
gradient  and  curvature  drifts  drift  closer  to  the  earth  than  those  with  higher  energies.  The 
boundary  (termed  ble)  demarcates  the  most  equatorward  location  from  which  measured 
energies  begin  to  increase,  which  corresponds  to  the  field  lines  threading  the  most 
earthward  regions  of  the  plasma  sheet.  Its  geophysical  meaning  is,  therefore,  associated 
with  the  zero-energy  Alfven  layer  and  the  plasmapause  (Newell  et  ah,  1996,  2002).  The 
existence  of  this  boundary  can  be  conceptualized  using  a  DMSP  satellite  data 
spectrogram,  shown  in  Figure  10.  The  ble  boundary,  delineated  by  the  black  line, 
corresponds  to  the  onset  of  low-energy  electron  energy  fluxes  that  occurs  just  after  00:27 
UT.  As  was  discussed  in  Figure  6,  the  energy  fluxes  initially  increase  as  a  function  of 
latitude  just  inside  of  the  equatorward  boundary. 
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Figure  10.  Sample  DMSP  spectrogram  obtained  on  August  30,  2006.  The  ble  boundary  (thick  black 
line)  occurs  at  approximately  00:27:30  UT  corresponding  to  the  onset  of  low-energy  electrons. 
Spectrogram  obtained  from  JHU/APL  website:  http://sd-www.jhuapl.edu/Aurora/spectrogram/ 


At  some  point  after  the  onset  of  particle  precipitation,  however,  the  energy  flux 
measurements  plateau  within  the  auroral  oval.  This  occurs  because  the  particle  energies 
found  within  the  main  plasma  sheet  have  a  much  more  Maxwellian  distribution  that  those 
that  have  migrated  earthward.  The  point  at  which  the  change  in  the  average  energy  no 
longer  increases  with  latitude  corresponds  to  Newell’s  b2e  boundary.  This  boundary  is 
topologically  associated  with  the  most  earthward  extent  of  the  main  body  of  the  plasma 
sheet. 

These  two  boundaries  are  useful  in  the  development  of  this  study.  The  region 
between  the  ble  and  b2e  boundaries  clearly  demarcates  the  equatorward  extent  of  auroral 
precipitation,  but  various  camps  support  utilizing  one  or  the  other  (Newell  et  ah,  2002). 
This  study  undertaken  here  has  been  designed  to  rank  how  accurately  various  models 
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depict  the  location  of  the  equatorward  boundary  of  the  auroral  oval  using  carefully 
chosen,  fixed  energy  flux  values  to  designate  the  effective  boundary.  The  information 
gained  from  Newell’s  taxonomic  research  merits  and  justifies  using  a  boundary  criterion 
that  is  neither  the  ble  nor  the  b2e  boundary  but,  instead,  corresponds  to  an  energy  flux 
value  statistically  certain  to  fall  between  the  two  boundaries. 

2.3  Auroral  Categorization  and  Measurements 

Advancements  in  computing  technology  allow  today’s  auroral  models  to  ingest, 
process,  and  compile  large  amounts  of  space  weather  data  from  a  multitude  of  sources. 
However,  despite  these  advances,  the  Kp  index  remains  one  of  the  most  valuable  means 
of  categorizing  auroral  activity.  A  brief  summary  of  how  this  index  is  determined  and 
utilized  will  facilitate  later  discussion. 

In  Section  2.3.2,  the  basic  operation  of  the  DMSP  satellites  will  be  discussed. 
DMSP  has  been  providing  polar  region  data  for  decades  and  has  become  one  of  the  most 
prolific  sources  of  data  in  the  near-earth  environment.  Because  DMSP  data  is  integral  to 
this  research,  some  basic  information  will  be  provided  to  establish  familiarity  which  will 
be  relied  upon  in  the  remainder  of  this  document. 

2.3.1  The  Planetary  K  Index 

The  planetary  K  (Kp)  index,  fielded  in  1949,  remains  a  prevalent  modeling 
parameter  to  this  day.  It  measures  geomagnetic  activity  on  a  scale  from  0  to  9  based 
upon  measurements  taken  from  a  network  of  13  ground  stations.  Transient  fluctuations 
in  the  magnetic  field  at  the  earth’s  surface  are  induced  by  changes  in  the  ionospheric  and 
magnetospheric  currents  (e.g.,  the  ring  current),  which  vary  with  changes  in  geomagnetic 
activity  and  solar  wind  conditions  (Menvielle  &  Berthelier,  1991).  The  Kp  index  is 
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determined,  in  part,  by  measuring  the  magnitude  of  the  surface  field  fluctuations.  It  is 
particularly  suitable  for  auroral  studies  because  variations  in  the  earthward  boundary  of 
the  plasma  sheet  have  been  shown  to  correlate  well  to  the  Kp  index  value  (Wing,  2005). 
Because  of  its  ubiquity  and  uniformity,  the  Kp  index  will  be  used  extensively  to 
categorize  data  sets  within  this  study. 

2.3.2  DMSP  Satellites  and  Sensors 

The  Defense  Meteorological  Satellite  Program  (DMSP)  launched  its  first  satellite 
in  1963,  and  the  program  remains  in  service.  Each  DMSP  satellite  has  a  nearly  polar, 
sun-synchronous,  -101  minute  orbit  at  -830-850  km.  The  polar  orbits  pennit  each 
satellite  to  transit  through  the  auroral  ring  twice  in  each  hemisphere  (once  going 
northward  and  once  going  southward).  Optimally,  as  many  as  four  satellites  are  in 
operation  at  the  same  time. 

In  the  mid-1970s,  specially  designed  particle  detectors  were  installed  on  the 
DMSP  satellites  to  measure  the  number  and  energy  fluxes  of  precipitating  electrons  and 
ions.  The  latest  in  the  series  of  sensors  is  the  SSJ/4,  an  ion  and  electron  spectrometer  that 
is  oriented  on  the  satellites  such  that  it  always  faces  the  local  zenith.  The  sensors  collect 
data  at  one-second  intervals  and  use  electrostatic  analyzers  to  measure  precipitating 
electron  and  ion  energy  fluxes  (eV/cm  /s/sr)  in  one  of  20  logarithmically  spaced  energy 
channels  between  30  eV  and  30  keV  (Redmon  et  ah,  2010).  The  DMSP  satellites  provide 
one  of  the  best  space-based  platforms  for  measuring  the  precipitating  ions  and  electrons 
while  reducing  exposure  to  back-scattered  radiation  (Gussenhoven  &  Hardy,  1983).  The 
spectrogram  in  Figure  10  is  one  popular  graphical  representation  of  the  data  obtained 
from  the  satellites.  In  2006,  an  upgraded  SSJ/5  sensor  was  installed  on  DMSP  satellite 
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FI  7.  The  SSJ/5  sensor’s  capabilities  are  enhanced,  but  for  purposes  of  this  study,  its  data 
is  indistinguishable  from  the  SSJ/4  data. 

2.4  Auroral  Modeling 

The  development  of  more  space-based  platforms  has  introduced  many  different 
ways  to  observe  and  measure  auroral  activity  (Siscoe,  1991).  Some  platforms,  like 
NASA’s  Polar  satellite,  take  synoptic  measurements  and  are  useful  in  performing  large- 
scale  measurements  like  hemispheric  auroral  power.  As  was  just  discussed  in  a  previous 
section,  the  low  orbiting  DMSP  satellites  continue  to  collect  extensive  amounts  of 
particle  and  energy  data  with  high  spatial  resolution.  Other  platforms,  located  well 
beyond  earth’s  magnetosphere,  directly  measure  ambient  solar  wind  and  IMF  parameters. 
These  measurements  (e.g.,  solar  wind  speed,  IMF  orientation,  etc.)  have  become  an 
integral  part  of  the  computations  done  by  more  advanced  precipitation  models.  The 
Advanced  Composition  Explorer  (ACE)  satellite,  located  approximately  1.5M  km  from 
earth,  is  an  example  of  a  platform  that  obtains  this  type  of  data.  Each  of  these  platfonns 
has  enhanced  the  ways  auroral  activity  can  be  observed.  However,  the  addition  of  more 
and  more  types  of  data  still  requires  the  modelers  to  carefully  choose  how  to  use  this  data 
in  order  to  create  the  most  accurate  results.  This,  in  general,  has  proved  to  be  a 
formidable  challenge. 

The  data  limitations  that  beset  the  earliest  models  meant  they  were  usually 
developed  from  statistical  studies  of  empirical  data.  A  better  theoretical  understanding 
has  allowed  more  complex  processes  to  be  introduced,  but  most  researchers  agree  that  a 
completely  physics-driven  model  is  not  imminent.  The  five  models  included  in  this  study 
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span  several  decades  of  advancement  and  understanding.  The  following  sections  will 
provide  some  basic  background  information  on  each  of  them. 

2.4.1  The  Hardy  Model  -  1985  Version 

Data  obtained  between  1977  and  1980  from  more  than  27,000  satellite  passes  of  3 
DMSP  satellites  was  utilized  in  1985  to  create  what  has  become  one  of  the  most  well- 
known,  empirical  auroral  models — Hardy  and  Gussenhoven’s  Statistical  Model  of 
Electron  Precipitation  (Hardy  et  ah,  1985).  Hereafter,  for  simplicity,  this  model  will  be 
referred  to  as  the  Original  Hardy  (OH).  The  OH  model  depicts  the  global  pattern  of 
electron  precipitation  in  the  polar  region  (in  1987,  it  was  modified  to  include  ion 
precipitation  as  well)  as  a  function  of  MLT,  MLAT,  and  Kp. 

The  model  was  developed  by  perfonning  a  comprehensive  statistical  study.  The 
polar  ionosphere  was  divided  into  a  grid  of  1440  sections  comprised  of  48  MLT  bins 
(30-minute  sections)  and  30  MLAT  bins  between  50°-90°  (1°  increments  between 
60°-80°  and  2°  increments  elsewhere).  Seven  individual  grids  were  then  assembled 
based  upon  grouping  Kp  index  values  (Hardy  et  al.,  1985).  The  seven  groupings  are 
listed  in  Table  1.  Hardy’s  study  used  approximately  14.1  million  individual  spectra, 
whose  Kp  distribution  percentages  are  shown  in  Table  2.  Group  7,  despite  comprising  9 
discrete  Kp  levels,  is  the  group  with  the  smallest  population  of  observed  events. 
Extremely  high  Kp  events  are  quite  rare,  and  this  poses  a  challenge  for  both  the  models 
and  the  studies  designed  to  validate  them. 
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Table  1.  Original  Hardy  model  Kp  grouping  (Hardy  et  al.,  1985). 


Group 

Kp  Values 

1 

0,  0+ 

2 

1-,  1,  1+ 

3 

2-,  2,  2+ 

4 

3-,  3,  3+ 

5 

4-,  4,  4+ 

6 

5-,  5,  5+ 

7 

6-,  6,  6+,  7-,  7,  7+,  8-,  8,  8+,  9-,  9 

Table  2.  Original  Hardy  model  Kp  distribution  of  analyzed  spectra  (Hardy  et  al.,  1985). 


Group 

Percent  Observed 

1 

8.0 

2 

23.8 

3 

26.9 

4 

21.9 

5 

10.5 

6 

5.3 

7 

3.6 

In  each  grid  element,  the  average  and  standard  deviation  of  the  differential 
number  flux  in  each  of  the  discrete  energy  channels  of  the  SSJ  detector  was  used  to 
calculate  an  average  differential  number  flux  spectrum  for  each  group  of  Kp  indices.  To 
facilitate  mapping  the  results,  the  integral  number  flux  (cm  /s/sr)"  and  the  energy  flux 
(keV/cm  /s/sr)  were  then  determined,  based  upon  the  differential  number  flux  and  the 
centerline  energy  of  a  respective  energy  bin.  Noise  was  eliminated  by  using  a  smoothing 
function  and  averaging  the  values  in  contiguous  grid  elements  three  times  (Hardy  et  al., 
1985).  Each  computed  value  was  mapped  to  a  polar  plot  for  a  given  Kp  level.  The  result 
is  seven  polar  maps,  one  for  each  Kp  group,  displaying  number  flux,  energy  flux,  or 
average  energy  in  each  MLT-MLAT  grid  element. 
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Figure  1 1  displays  the  energy  flux  output  for  quiet  (Kp  1)  and  stormy  (Kp  6) 
conditions  respectively.  An  interpolating  function  has  been  applied  to  reduce  the  MLAT 
grid  size  to  0. 1°  versus  0.5°.  These  plots  clearly  depict  the  significant  changes  in  total 
energy  and  spatial  extent  of  the  auroral  zone  precipitation  during  different  conditions. 
During  high-Kp  conditions,  the  expansion  of  the  equatorward  boundary  of  the  oval  is 
readily  apparent. 


Figure  11.  OH  model  polar  plots  showing  Kp  1  (left)  and  Kp  6  (right). 


2.4.2  The  OVATION  Prime  Model 

One  significant  simplification  of  the  Hardy  model  is  that  it  does  not  distinguish 
between  particles  whose  energy  spectra  are  quite  disparate.  Different  precipitation 
regions  move  about  continually,  therefore,  averaging  at  fixed  MLAT-MLT  locations 
mixes  the  different  types  together  (Sotirelis  &  Newell,  2000). 

Distinguishing  among  different  particles  was  one  tenet  of  the  OVATION  Prime 
(OP)  model,  released  by  Newell  and  colleagues  in  2009.  This  model  parameterizes 
auroral  electrons  into  three  distinct  types  and  separately  accounts  for  ions.  Two  electron 
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classifications  account  for  accelerated  particles.  Newell  names  these  monoenergetic 
energy  events,  attributed  to  electric  field  acceleration,  and  broadband  acceleration, 
produced  by  Alfven  wave  dispersion.  Any  electrons  not  meeting  either  of  the  two 
previous  definitions  are  classed  as  diffuse.  This  model  does  not  make  any  distinction 
among  ions,  but  it  does  consider  them  separately  from  electrons  (Newell  et  al.,  2009). 

Electrons  are  categorized  by  assessing  DMSP  energy  flux  data  which  enables  the 
model  to  distinguish  between  the  two  types  of  electron  acceleration.  The  DMSP 
satellite’s  SSJ/4  sensor  measures  energy  flux  in  each  of  20  discrete  bins.  For  a 
monoenergetic  event,  the  energy  flux  must  drop  to  30%  of  the  maximum  of  the  peak 
channel  measurement  within  two  energy  steps  above  or  below  the  peak.  Contrarily, 
broadband  acceleration  is  categorized  when  three  or  more  channels  have  energy  flux 
measurements  above  2.0  x  10'  eV/cm  /s/sr/eV  and  at  least  one  of  the  channels  satisfying 
this  criterion  occurs  in  an  energy  channel  exceeding  140  eV  (Newell  et  al.,  2009). 

The  OP  model  also  maps  to  a  polar  MLT-MLAT  grid.  The  model’s  resolution  is 
0.25°  MLT  and  0.50°  MLAT  between  50°-90°.  In  each  grid  element,  energy  flux  is 
computed  (erg/cm  /sec),  but  the  output  is  not  discretely  parameterized  by  Kp  index. 
Instead,  the  OP  model  is  parameterized  by  solar  wind  driving,  which  its  researchers 
believe  is  best  represented  by  the  following  solar  wind  coupling  function,  d<pMP/ dt, 
magnetic  flux  at  the  magnetopause.  This  equation  is  given  by 

^  =  v'k  •  S%  .  sin%  (|)  (6) 

where  v  is  bulk  solar  wind  velocity  (km/s),  B  is  ( By  +  Bf)  ' 2  (nT)  and  6  is  IMF  clock 
angle.  The  solar  wind  data  comes  from  the  OMNI2  data  set  supplied  by  NASA  Goddard. 
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The  0MNI2  data  set  contains  solar  wind  data  obtained  from  a  variety  of  satellites  and  is 
tabulated  once  per  hour.  Timing  is  reconciled  by  using  data  pre-propagated  to  ~10  Re, 
the  approximate  location  the  bow  shock  (Newell  et  ah,  2010b).  The  reconciliation  is 
necessary  because  data  concerning  the  solar  wind  incident  on  the  earth’s  magnetosphere 
is  often  measured  by  a  satellite  far  upstream  from  the  magnetopause  (e.g.,  -225  RE). 

This  solar  wind  coupling  function  serves  as  an  organizing  parameter  for  the 
model,  and  the  statistical  analysis  is  based  upon  least  squares  regression  of  the  form 

Auroral  Power  (  ^MP\  —  a  +  b  ■  ^MP  (7) 

V  at  )  at 

where  the  auroral  power  is  calculated  in  each  grid  element.  There  are  46,080  individual 
regression  fits  which  constitute  the  model  (4  categories  x  120  ML  AT  bins  x  96  MLT 
bins).  The  model  then  calculates  the  energy  flux  in  a  specific  MLT-MLAT  bin  by  taking 
the  product  of  the  fitted  estimate  of  the  auroral  intensity  and  the  probability  of  observing 
the  specific  type  of  aurora.  Seasonal  variations  in  each  of  the  four  types  of  auroral 
particles  are  also  considered.  A  season  is  defined  as  a  90-day  period  centered  on  the 
respective  equinox  or  solstice  (Newell  et  al.,  2010b).  The  seasonal  dependence  accounts 
for  different  energy  input  into  the  ionosphere.  For  a  given  solar  wind  condition, 
hemispheric  energy  flux  in  winter  is  increased  by  a  factor  of  1.18-1.30  over  summer 
(Newell  et  al.,  2010b). 

One  assumption  the  OP  model  makes  is  that  all  types  of  aurora  are  isotropic. 

Thus,  all  flux  values  are  multiplied  by  n  to  eliminate  any  directional  dependence.  This 
assumption  is  both  reasonable  and  justified,  but  it  may  slightly  overestimate  broadband 
and  monoenergetic  auroral  fluxes,  which  tend  to  be  field  aligned  (Newell  et  al.,  2009).  In 
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a  given  MLAT-MLT  grid  element,  the  total  energy  flux  is  simply  the  sum  of  each  of  the 
fluxes  calculated  for  the  four  types  of  auroral  particles  described  above. 

For  the  purposes  of  this  study,  only  total  flux  will  be  considered.  The  polar  plots 
in  Figure  12  provide  two  examples  of  the  OP  model’s  output  during  different 
geomagnetic  conditions.  The  general  shape  of  the  auroral  oval  is  relatively  easy  to 
discern  in  both  plots,  particularly  between  dusk  and  dawn.  For  periods  of  heightened 
geomagnetic  stonning,  the  model  produces  a  larger  and  more  energetic  auroral  zone. 
Very  large  energy  flux  values  are  observed  in  the  night  hemisphere,  and  the  equatorward 
boundary  has  expanded  such  that  it  is  well  below  60°  MLAT. 


Figure  12.  OP  polar  plots  showing  quiet  solar  driving  (left)  versus  storming/Kp  6+  (right). 


2.4.3  The  Hardy  Model  -  2008  Adaptation 

In  2008,  Hardy  and  colleagues  reinvestigated  their  study  of  electron  precipitation 
at  high  latitudes.  Like  the  original  research  from  1985,  this  study  conducted  a  statistical 
analysis  of  DMSP  data,  only  now  more  than  600  million  individual  spectra  were  used 
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(Hardy  et  al,  2008).  This  model  distinguishes  itself  by  using  probability  distributions  to 
characterize  different  particle  populations  in  lieu  of  average  energy  flux  values. 

This  adaptation  to  the  OH  model  was  motivated  by  the  perception  of  an  inherent 
problem  in  the  way  the  OH  model  assumption  that  all  particle  populations  represent 
unimodal,  normal  distributions.  Statistically  speaking,  this  assumption  implies  the 
average  value  corresponds  to  the  most  probable  (Hardy  et  al.,  2008).  However,  the 
various  regions  of  the  magnetosphere  migrate  within  the  magnetosphere  as  functions  of 
geomagnetic  activity,  and  as  a  result,  the  connection  between  the  ionosphere  and 
magnetosphere  along  the  geomagnetic  field  lines  also  changes.  Thus,  particles 
indigenous  to  specific  regions  of  the  magnetosphere  may  under  certain  conditions 
precipitate  into  different  regions  of  the  ionosphere,  and  taking  simple  averages  may  lead 
to  the  mixing  of  precipitating  particle  characteristics.  The  researchers  argue  the  final 
results  may  not  represent  characteristics  associated  with  any  of  the  original  source 
particle  populations. 

In  the  new  study,  more  highly  refined  analysis  of  the  DMSP  data  confirmed  that 
the  probability  distributions  of  electrons  within  a  specific  grid  element  were  always 
lognonnal  distributions  and  often  multimodal  (Hardy  et  al.,  2008).  This  consideration 
was  factored  into  the  development  of  new  precipitation  patterns,  whose  modeling 
enhancements  will  be  referred  to  as  the  New  Hardy  (NH)  model  in  this  study. 
Comparisons  between  OH  and  NH  output  indicate  a  similar  structure  with  large  areas  of 
agreement.  However,  Hardy  et  al.  (2008)  demonstrated  there  is  a  less  than  10% 
probability  that  the  particle  precipitation  in  any  particular  MLAT-MLT-Kp  grid  element 
satisfies  the  unimodal  assumption. 
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The  NH  model  adjusts  the  OH  model’s  output  based  upon  this  statistical 
conclusion  that  in  a  given  MLT-MLAT-Kp  grid  element,  there  is  a  low  probability  of  the 
average  integral  flux  coinciding  with  the  most  probable.  Figure  13  shows  two  examples 
of  synoptic  plots  of  the  NH  model’s  energy  flux  calculations  during  quiet  and  stormy 
conditions.  Again,  an  interpolating  function  allows  a  0.1°  MLAT  resolution.  The 
model’s  equatorward  boundary  during  stonny  conditions  does  not  exhibit  the  typical 
bulging  near  the  midnight  MLT  sectors,  and  generally  the  plot  is  less  energetic  than  was 
found  with  the  OH  model. 


Figure  13.  NH  polar  plot  showing  Kp  1  (left)  and  Kp  6  (right). 

2.4.4  Space  Weather  Modeling  Framework  Model 

The  entire  Space  Weather  Modeling  Framework  (SWMF)  consists  of  10  major 
components  representing  the  entire  domain  of  solar-terrestrial  space  weather.  The  entire 
SWMF  chain  is  maintained  at  NASA-Goddard.  A  method  to  calculate  auroral 
precipitation  patterns  requires  coupling  the  Global  Magnetosphere  (GM)  component  of 
the  SWMF  to  the  Fok  Ring  Current  (RC)  model. 
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In  this  process,  the  SWMF/GM  model  provides  a  physics-based 
magnetohydrodynamic  (MHD)  characterization  of  the  earth’s  magnetosphere  (Toth  et  ah, 
2005).  The  Fok  RC  model  generates  a  kinetic  description  of  the  global  particle 
distribution  in  the  inner  magnetosphere  in  the  energy  range  of  1-300  keV.  By  taking 
advantage  of  the  dynamic  electric  and  magnetic  fields  modeled  by  SWMF/GM,  the  Fok- 
RC  model  is  capable  of  computing  the  equatorward  boundary  of  auroral  precipitation 
patterns  (Zheng,  2012). 

The  dynamics  of  the  ring  current  have  been  demonstrated  to  be  closely  tied  to 
geomagnetic  stonning.  The  ring  current  flows  in  a  westerly  direction  in  the  near-earth 
region,  between  3  and  6  earth  radii.  Particles  in  the  current  region  are  lost  as  a  result  of 
pitch  angle  diffusion  and  subsequent  collisions  in  the  upper  atmosphere  in  the  mid  and 
high  latitudes  (Prolss,  2004). 

For  simplicity,  this  model  will  be  referred  to  as  the  SWMF  model.  The  SWMF 
model  assumes  30%  of  the  particle  fluxes  crossing  the  equatorial  plane  are  scattered  into 
the  loss  cone,  where  they  precipitate  into  the  ionosphere.  Utilizing  characteristic  energy 
distributions  found  in  the  ring  current  region,  an  energy  flux  is  obtained.  The  output  of 
the  model  is  the  most  equatorward  location  at  which  a  specified  energy  flux  value  is 
obtained.  Its  longitudinal  resolution  is  coarse,  limited  to  one  hour  MLT  spacing.  At  this 
time,  plotting  an  entire  polar  grid  is  not  possible  with  the  model’s  output.  A 
representative  sample  of  a  boundary  provided  by  the  SWMF  model  is  shown  in  Figure 
14.  Generally  speaking,  the  SWMF’s  boundary  is  highly  symmetric  about  the  magnetic 
pole  and  largely  invariant  to  changes  in  Kp.  This  impact  of  this  invariance  on  the 
model’s  comparative  performance  will  be  discussed  in  greater  detail  in  Chapter  IV. 
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Figure  14.  SWMF  characteristic  boundary  plot  during  Kp  3-.  Boundary  corresponds  to  an  energy 
flux  threshold  of  0.4  erg/cm2/s. 

2.4.5  Assimilative  Mapping  of  Ionospheric  Electrodynamics 

The  Assimilative  Mapping  of  Ionospheric  Electrodynamics  ( AMIE)  procedure  is 
a  specification  model  that  was  fielded  in  1988.  Its  purpose  is  to  synthesize  collections  of 
diverse  data  relating  to  the  high-latitude  ionospheric  environment  into  patterns  of 
conductivities,  electric  fields,  and  currents  (Richmond  et  ah,  1998).  The  AMIE  model 
processes  all  data  related  to  ionospheric  conductances,  including  any  observed  data 
involving  the  energy  fluxes  of  auroral  particles,  to  obtain  a  conductance  model.  The 
conductance  model  can  be  used  to  estimate  the  energy  flux  of  auroral  particles 
throughout  the  polar  regions.  One  of  the  capabilities  for  which  AMIE  was  specifically 
designed  pertains  to  auroral  conductance  and  precipitation  patterns. 
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As  a  specification  model,  AMIE  procedure  incorporates  data  from  ground-  and 
space-based  instruments,  including  DMSP  particle  flux  data.  The  model  output  is 
generated  by  performing  an  optimized,  weighted  least-squares  fit  of  empirically 
measured  parameters  to  existing  statistical  models.  In  effect,  AMIE  extrapolates  as  much 
information  as  possible  from  existing  measurements  and  then  utilizes  fitting  functions  to 
fill  in  where  data  is  absent  (Richmond  et  al.,  1998).  The  model’s  accuracy  is  inversely 
related  to  the  amount  of  empirical  data  it  is  able  to  ingest,  but  the  model  can  be  used  in  a 
purely  statistical  sense,  if  there  is  no  empirical  data  available. 

This  model  will  be  referred  to  as  the  AMIE  model.  The  data  obtained  for  this 
study  allowed  for  a  model  output  with  very  good  temporal  resolution,  but  the  data  was 
limited  to  a  spatial  resolution  of  1  hour  MLT.  Figure  15  depicts  the  result  of  an  AMIE 
run  for  Kp  5  conditions.  The  reduction  in  spatial  fidelity  is  readily  apparent,  and  it 
becomes  more  problematic  at  more  equatorward  latitudes  due  to  the  expansion  of  the  grid 
elements  as  a  function  of  ML  AT. 


Figure  15.  AMIE  polar  plot  for  Kp  5  conditions.  Its  coarse  resolution  makes  equatorward  boundary 
identification  difficult. 
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III.  Methodology 


This  chapter  describes  the  methods  used  to  acquire  and  analyze  the  data  utilized 
in  this  study.  The  first  section  pertains  to  the  DMSP  data  and  includes  the  means  by 
which  the  data  was  obtained  and  processed.  Then,  the  different  methods  used  to  calculate 
each  model’s  equatorward  boundary  are  detailed.  Finally,  the  statistical  measures  that 
were  used  to  establish  meaningful  comparisons  between  the  models  and  the  DMSP  data 
are  explained. 

3.1  DMSP  Data 

DMSP  data  was  obtained  from  an  open-source  website  maintained  by  the  Auroral 
Particles  and  Imagery  Group  at  the  Johns  Hopkins  University  Applied  Physics 
Laboratory  (JHU/APL).  A  client-based  JAVA  application  allowed  the  DMSP  data  to  be 
retrieved  and  downloaded.  Data  was  obtained  from  each  satellite  on  a  per  day  basis. 

3.1.1  DMSP  Data  Acquisition 

Table  3  is  an  extract  from  a  data  file  containing  one  second  of  data  of  data 
recorded  by  DMSP  Satellite  F13  on  May  15,  2005.  The  UTC  column  includes  the  date 
and  time  stamp  of  the  data  entry  (seconds  are  not  displayed).  The  GLAT  (degrees)  and 
GLON  (degrees)  columns  convey  the  geographic  latitude  and  longitude  coordinates  of  the 
satellite  in  orbit.  The  MLAT  (degrees)  coordinate  depicts  the  satellite’s  position  using 
geomagnetic  coordinates,  and  the  MLT (hours)  entry  describes  the  longitudinal 
coordinate  of  the  satellite’s  position.  The  MLT  coordinate  maintains  a  fixed  reference  to 
the  sun,  such  that  12  MLT  meridian  is  always  sub-solar. 
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JEe  and  JEi  entries  represent  the  integral  energy  flux  (eV/cm  /s/sr)  for  the 
electrons  and  ions  respectively,  and  the  AvgEe  and  AvgEi  represent  the  average  energy 
(eV)  per  electron  and  ion. 


Table  3.  DMSP  satellite  data  taken  from  Satellite  F13  on  May  15,  2005. 


UTC 

GLAT 

GLON 

MLAT 

MLT 

JEe 

AvgEe 

JEi 

AvgEi 

5/15/2005  0:01 

81.37 

225.90 

82.22 

11.28 

3.54E+09 

1.59E+02 

1.86E+09 

6.20E+03 

Although  the  SSJ/4  sensor  captures  energy  data  in  each  of  the  20  discrete 
channels,  only  the  integral  energy  value  was  used  in  this  study.  Furthermore,  this  study 
made  no  distinction  between  electron  and  ion  aurora  with  regard  to  the  equatorward 
extent  of  the  oval.  The  oval  was  determined  based  upon  measurements  of  the  total 
integral  energy  flux,  calculated  by  adding  the  electron  (JEe)  and  ion  (JEi)  energy  fluxes 
together. 

All  of  the  events  included  in  this  study  occurred  between  2000-2008.  The  first 
dates  selected  for  this  study  aligned  with  five  prominent  geomagnetic  stonns.  This  list 
included  13  total  days,  grouped  into  two-,  three-,  and  four-day  long  events.  The  span  of 
days  for  each  event  included  the  buildup  and  recovery  phases  of  the  storm.  After  data  for 
these  days  was  obtained,  an  additional  1 5  days  were  then  added  to  include  more  low  and 
moderate  Kp  index  values  and  to  represent  a  more  diverse  selection  of  months  and  years. 
A  list  of  dates  is  included  in  Table  4. 

Data  was  obtained  from  three  satellites  on  each  of  the  days  in  the  study.  The 
SSJ/5  sensor  is  installed  on  Satellite  FI 7,  but,  as  was  mentioned  previously,  it  provides 
the  same  integral  energy  flux  data  as  the  SSJ/4  sensor  for  the  purposes  of  this  study. 
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Table  4.  List  of  dates  investigated. 


Dates 

Days 

DMSP  Sat  No. 

Prominent 

Storms 

08/31/01  -  09/01/01 

2 

14,  15,  16 

05/15/05  -  05/16/05 

2 

13, 15,  16 

07/09/05  -  07/12/05 

4 

13,  15,  16 

08/31/05  -  09/01/05 

2 

13,  15,  16 

12/14/06  -  12/16/06 

3 

13,  15,  16 

Additional 

02/23/00  -  02/25/00 

3 

13,  14,  15 

10/23/02  -  10/25/02 

3 

13,  14, 15 

10/12/04  -  10/14/04 

3 

13, 15,  16 

06/19/07  -  06/21/07 

3 

13, 15,  17 

03/26/08  -  03/28/08 

3 

15,  16,  17 

3.1.2  DMSP  Data  Compilation 

The  Matrix  Laboratory  (MATLAB)  program  was  used  to  process  all  the  DMSP 
data.  Each  of  the  DMSP  data  files  was  read  into  a  specially  developed  MATLAB 
algorithm.  This  algorithm  interrogated  the  energy  flux  data  and  automatically  determined 
the  MLAT-MLT  coordinate  of  the  equatorward  boundary.  Then,  the  location  of  the 
boundary  was  plotted  on  a  geomagnetic  polar  grid. 

Up  to  14  orbits  occurred  during  a  24  hour  period;  however,  there  were  several 
conditions  disqualified  the  orbit  from  the  study.  If  the  beginning  of  the  data  file 
corresponded  to  the  satellite  orbiting  within  the  polar  region,  then  that  particular  orbit 
was  discarded.  There  were  also  multiple  instances  in  the  DMSP  data  files  were  no 
energy  flux  data  recorded.  These  data  points,  identified  by  entries  were  the  energy  flux 
value  was  0.0  eV/cm  /s/sr,  were  deemed  nonphysical  and  removed  from  the  data  file 
before  processing  it.  If  more  than  10%  of  the  entries  in  a  data  file  recorded 
0.0  eV/cnr/s/sr,  then  the  entire  data  file  was  deleted  and  another  DMSP  satellite  was 
chosen. 
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Once  the  steps  presented  in  the  previous  paragraph  were  used  to  condition  the 
data  fde,  a  15-second,  moving  average  of  the  total  energy  flux  was  calculated.  Using  a 
moving  average  was  an  effective  way  to  mitigate  the  impact  of  other  spurious  data  points. 
Due  to  the  satellites’  trajectories,  applying  this  average  introduced  approximately  0.75° 
of  uncertainty  in  the  MLAT  coordinate  but  eliminated  a  large  number  of  false  crossing 
points. 

In  this  study,  the  effective  location  of  the  equatorward  boundary  of  the  auroral 
oval  was  defined  as  the  most  equatorward  MLAT  where  the  average  total  integral  energy 
flux  exceeded  0.4  erg/cm  /s.  Another  set  of  boundaries  was  determined  using  a 
0.6  erg/cm  /s  threshold,  and  this  data  set  was  used  to  corroborate  the  results  found  at  the 
lower  flux  threshold.  As  was  discussed  in  Chapter  II,  no  commonly  accepted  value  can 
be  said  to  perfectly  represent  the  boundary.  These  quantities  were  selected  because  they 
were  above  the  background  readings  of  the  detector  but  below  the  typical  maximum 
readings  observed  inside  the  auroral  oval.  Because  the  DMSP  satellites  measure 
directional  flux  in  units  of  eV/cm  /s/sr,  the  energy  fluxes  were  divided  by  n  to  obtain  an 
equivalent  value.  Table  5  lists  the  unit  conversions  for  each  threshold. 


Table  5.  Threshold  Fluxes  (ener/cm2/s) 


erg 

eV 

eV  (per  sr) 

0.4 

2.5  xlO11 

8.0  xlO10 

0.6 

3.7  x  10U 

1.2  x  1011 

1 

6.2  xlO11 

2.0  x  1011 

The  plot  in  Figure  16  graphically  depicts  how  the  equatorward  boundary  of  each 
orbit  was  determined.  Each  blue  dot  represents  an  energy  flux  measurement  taken  by  the 
sensor.  The  heavy  black  line  represents  the  15 -second  moving  average  of  the  energy  flux 
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measurements.  The  thin  horizontal  line  is  the  energy  flux  threshold.  The  effective 
equatorward  boundary  occurs  at  the  location  where  the  heavy  black  line  intersects  the 
threshold.  As  was  previously  mentioned,  each  DMSP  data  file  contained  as  many  as  14 
orbits.  In  a  polar  orbit,  a  northbound  and  a  southbound  crossing  can  be  obtained, 
resulting  in  up  to  28  total  crossings  per  satellite  per  day. 
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Figure  16.  DMSP  orbit  plot  taken  from  DMSP  Satellite  F15  on  August  31,  2001  during  low  Kp 
conditions.  Energy  flux  threshold  is  8.0xl010  eV/cm2/s/sr.  Red  arrow  indicates  false  boundary  if  no 
smoothing  function  is  used.  The  circled  green  X  indicates  the  location  of  the  effective  boundary. 


These  plots  were  used  to  validate  the  location  of  the  boundary  coordinates. 
Despite  using  the  smoothing  function,  occasionally  a  single,  high-energy  data  point  was 
able  to  foul  the  moving  average.  An  example  of  this  is  shown  in  Figure  17.  These 
occurrences  were  also  deemed  non-physical,  and  the  boundary  coordinate  was  excluded 
from  the  study.  To  ensure  the  highest  standard  of  date  integrity,  a  plot  was  created  for 
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every  boundary  and  was  visually  inspected  for  anomalies  like  the  one  shown  in  Figure 


17. 


Figure  17.  Example  of  discarded  orbit  due  to  anomalous  data. 

The  UTC  of  each  threshold  crossing  was  recorded,  and  the  crossings  that  occurred 
during  the  same  hour  were  grouped  together  and  plotted  on  a  polar  projection  map  like 
the  one  shown  in  Figure  18.  The  grid  is  divided  into  MLAT-MLT  sectors.  This  allowed 
chronological  crossing  points  to  be  plotted  on  the  same  map.  As  an  example,  Figure  18 
shows  the  location  of  the  six  threshold  crossings  found  between  0800-0900  UTC  on 
May  15,  2005,  during  high  Kp  conditions.  These  plots  were  originally  generated  to 
observe  the  qualitative  systematic  migration  of  the  equatorward  boundary,  but  they  also 
identified  the  considerable  variance  in  the  boundary  location  defined  by  DMSP  flux  data. 
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Figure  18.  DMSP  crossing  points  (blue  dots)  obtained  between  08-09  UTC  on  May  15,  2005  during 
high  Kp  period.  Crossing  points  determined  using  a  0.4  erg/cm2/s  energy  flux  threshold.  Six  data 
points  were  obtained  during  this  60  minute  period  from  the  F13,  F15,  and  F16  satellites,  whose 
trajectories  are  indicated  by  the  dashed  lines.  Grid  coordinates  are  MLAT-MLT. 

When  a  threshold  crossing  occurred,  the  UTC,  MLAT-MLT  coordinates,  and 
energy  flux  were  captured  from  the  DMSP  data.  Using  the  UTC,  the  corresponding  Kp 
index  value  was  obtained  from  a  separate  database  of  three-hour  Kp  index  values. 

After  a  day  of  satellite  data  was  run,  the  results  were  compiled  into  a  single  data 
file  for  each  day.  Table  6  contains  an  example  data  file.  This  table  lists  each  boundary 
obtained  during  a  24-hour  period.  There  is  a  missing  orbit  (4N)  near  6:00  UTC,  whose 
data  was  excluded  for  one  of  the  reasons  previously  discussed. 

This  data  was  used  to  initialize  the  model  runs.  The  OP,  SWMF,  and  AMIE 
models  required  the  UTC,  and  the  OH  and  NH  models  required  the  Kp  index  value. 

Based  on  this  input,  each  model  determined  a  precipitation  pattern,  and  this  output  was 
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interrogated  to  obtain  the  model’s  location  of  the  effective  equatorward  boundary.  The 
difference  between  the  DMSP  boundary  location  and  model’s  boundary  was  the 
fundamental  data  obtained  in  this  study.  The  process  used  to  obtain  each  model’s 
boundary  will  be  explained  in  the  next  section. 


Table  6.  DMSP  Threshold  Crossing  Data  obtained  from  DMSP  Satellite  F13  on  December  14,  2006. 
Missing  data  indicates  disqualified  orbit. 


Orbit 

UTC 

MLAT 

MLT 

Kp 

IN 

0:54 

73.27 

15.79 

3.0 

IS 

1:05 

67.79 

8.32 

3.0 

2N 

2:38 

73.67 

14.61 

3.0 

2S 

2:47 

67.34 

8.69 

3.0 

3N 

4:20 

69.55 

15.51 

2.3 

3S 

4:30 

66.44 

8.91 

2.3 

4N 

-- 

-- 

-- 

-- 

4S 

6:11 

68.62 

9.60 

2.0 

5N 

7:47 

74.90 

14.48 

2.0 

5S 

7:53 

70.20 

10.02 

2.0 

6N 

928 

72.82 

16.02 

2.0 

6S 

9:37 

68.23 

9.47 

2.0 

7N 

11:10 

74.33 

16.43 

2.0 

7S 

1 1:20 

68.83 

8.99 

2.0 

8N 

12:51 

73.42 

17.37 

5.0 

8S 

13:03 

67.09 

7.93 

5.0 

9N 

14:31 

68.79 

18.07 

5.0 

9S 

14:45 

63.82 

7.05 

5.0 

ION 

16:11 

63.65 

18.26 

5.3 

10S 

1626 

64.27 

6.58 

5.3 

1  IN 

17:51 

61.89 

18.37 

5.3 

1  IS 

18:07 

62.37 

6.52 

5.3 

12N 

19:33 

63.91 

18.52 

5.3 

12S 

19:47 

63.39 

6.81 

5.3 

13N 

21:14 

65.56 

18.40 

7.7 

13S 

2128 

64.54 

7.25 

7.7 

14N 

22:53 

61.38 

17.99 

7.7 

14S 

23:10 

62.86 

7.57 

7.7 
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3.2  Auroral  Precipitation  Models 

3.2.1  The  Original  Hardy  Model 

As  mentioned  in  Chapter  II,  the  original,  1985  version  of  the  Hardy  model  has 
been  designated  the  Original  Hardy  (OH)  model  in  this  study.  This  model  was  obtained 
from  Utah  State  University.  The  model’s  algorithm  generated  a  data  file  of  the  energy 
flux  values  at  each  MLAT-MLT  grid  element  (the  model’s  resolution  is  0.10°  MLAT  by 
0.25  hour  MLT).  Because  the  OH  model  is  parameterized  by  integral  Kp  index  value 
from  0  to  6,  only  seven  data  files  were  created. 

The  equatorward  boundary  of  the  OH  model  was  determined  by  interrogating 
each  MLT  sector  (4  sectors  per  hour)  from  low  to  high  MLAT  for  the  point  where  the 
flux  threshold  was  first  exceeded.  Figure  19  shows  the  results  for  Kp  values  of  1,  3,  and 
5  with  the  threshold  set  at  0.4  erg/cm  /s.  In  some  cases,  particular  near  12  MLT,  no 
effective  boundary  was  found  using  this  threshold. 


Figure  19.  OH  equatorward  boundaries  for  (a)  Kp  1,  (b)  Kp  3,  (c)  Kp  5.  Boundary  corresponds  to 
energy  flux  of  0.4  erg/cm2/s.  Absent  data  corresponds  to  MLT  sector  with  no  threshold  crossing. 


Then,  a  data  file  was  generated  that  compared  the  model  boundary  to  each  DMSP 
crossing  point.  To  do  this,  each  DMSP  MLT  coordinate  was  rounded  to  the  nearest  0.25 
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hour  MLT  (e.g,  10.13  was  rounded  to  10.25  MLT).  In  the  corresponding  MLT  wedge, 
the  model’s  equatorward  boundary  recorded.  The  signed  difference  between  the  DMSP 
MLAT  and  the  OH  MLAT  was  also  calculated.  A  positive  difference  indicated  the 
DMSP  boundary  was  found  more  toward  the  pole  (at  a  higher  MLAT)  than  the  model 
boundary. 

3.2.2  OVATION  Prime  Model 

Like  the  DMSP  data,  the  OP  model  was  also  accessed  at  the  Auroral  Particles  and 
Imagery  group  maintained  by  JHU/APL.  The  model’s  output  is  a  polar  map  very  similar 
to  OH,  except  it  is  not  parameterized  based  upon  a  single  input  condition.  To  generate  a 
plot,  the  OP  model  required  the  date  and  time  of  each  DMSP  crossing  event. 

The  source  data,  provided  by  Dr.  Patrick  Newell,  was  used  to  construct  an 
automatic  program  capable  of  running  the  model  without  using  the  web  interface.  The 
automated  program  took  an  input  file  consisting  of  dates  at  times  and  generated  an  output 
file  for  each  crossing. 

The  output  file  consisted  of  the  same  information  as  the  OH  model  contained; 
however,  the  OP’s  MLAT  resolution  was  0.50°  versus  0.10°.  Initially,  the  location  of  the 
boundary  was  detennined  in  the  same  was  as  was  for  OH,  but  OP’s  unusual  energy  flux 
calculations  at  low  latitude  led  to  many  false  boundaries.  Some  of  the  pixels  in  the 
annulus  between  50-60°  MLAT  were  found  have  very  high  energy  flux  values,  which 
generated  a  false  boundary  location.  To  mitigate  this  problem,  the  boundary  was  defined 
only  if  three  consecutive  energy  flux  values  in  an  MLT  wedge  were  above  the  threshold. 
If  this  criterion  was  met,  the  boundary  was  logged  at  the  first  of  the  three  MLAT 
coordinates.  Imposing  this  requirement  eliminated  most  of  the  false  boundary  triggers. 
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The  data  files  were  generated  the  same  way  they  were  with  the  OH  model:  each 
DMSP  crossing  MLT  was  rounded  to  the  nearest  0.25  hour,  and  the  MLAT  of  the 
model’s  equatorward  boundary  was  recorded  in  the  corresponding  MLT  wedge. 

Additionally,  to  demonstrate  the  systematic  characteristics  of  the  oval,  another 
MATLAB  program  was  developed  to  create  a  boundary  plot  of  the  full  oval.  An  example 
is  shown  in  Figure  20.  During  the  creation  of  these  plots,  more  anomalies  with  the 
model’s  boundary  were  observed,  in  spite  of  the  stricter  criteria  imposed.  An  example  of 
this  condition  is  shown  in  Figure  21. 


Figure  20.  OP  boundary  plot  generated  for  August  31,  2001  at  17:38  UTC  during  Kp  4  conditions. 
Boundary  corresponds  to  energy  flux  of  0.4  erg/cm2/s.  Absent  data  between  1130-1230  MLT  indicate 
sector  in  which  no  threshold  crossing  occurs. 
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Figure  21.  Example  of  spurious  OP  boundary  point  despite  imposing  stricter  selection  criteria.  Plot 
generated  for  August  31,  2001  at  00:25  UTC  during  quiet  conditions  (Kp  3-).  Boundary  in  2215 
MLT  sector  is  ~10°  displaced. 

3.2.3  The  New  Hardy  Model 

The  data  files  for  the  enhancements  to  the  original  Hardy  model  were  provided  by 
Dr.  Chin  Lin  and  colleagues  at  the  AFRL.  The  processing  of  this  data  followed  the  same 
approach  as  the  OH  model  did,  and  the  format  of  the  output  and  the  model’s  resolution 
were  identical.  Figure  22  is  the  NH  analog  of  Figure  19  and  presents  the  model’s 
boundary  location  at  each  Kp  index  values  of  1,  3  and  5.  If  no  data  point  exists,  it  means 
the  model  did  not  compute  an  energy  flux  value  above  the  threshold  in  that  MLT  sector. 
With  the  NH  model,  this  occurred  routinely  between  approximately  12-16  MLT. 

For  each  DMSP  crossing  point,  the  MLT  was  rounded  to  the  nearest  0.25  hour, 
and  the  MLAT  corresponding  to  the  most  equatorward  occurrence  of  an  energy  flux  at  or 
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above  the  threshold  was  recorded  into  a  data  file.  The  difference  between  the  two 
coordinates  was  also  determined  and  recorded. 


Figure  22.  NH  equatorward  boundaries  for  (a)  Kp  1,  (b)  Kp  3,  (c)  Kp  5.  Boundary  corresponds  to 
energy  flux  of  0.4  erg/cm2/s.  Absent  data  corresponds  to  MLT  sector  with  no  threshold  crossing. 

3.2.4  Space  Weather  Modeling  Framework  Model 

For  this  model,  researchers  at  NASA  CCMC  provided  a  file  with  boundary 
coordinates  only,  precluding  the  depiction  of  a  synoptic,  polar  plot.  Three  independent 
boundary  files,  however,  were  provided  for  each  of  the  three  energy  flux  thresholds  used 
in  this  study. 

One  file  was  available  for  each  day,  and  each  hour  was  divided  into  1 5  individual 
entries  spaced  between  3  and  4  minutes  apart.  Each  entry  contained  1  MLAT  coordinate 
in  each  of  the  24  MLT  sectors.  To  interrogate  this  data,  a  slightly  different  MATLAB 
algorithm  was  developed.  First,  for  each  DMSP  crossing  time  (UTC),  the  closest  SWMF 
data  entry  was  found.  Then,  the  DMSP  MLT  coordinate  was  rounded  to  the  nearest 
whole  hour,  and  the  model’s  corresponding  boundary  MLAT  was  recorded  into  a  file. 

The  difference  between  the  DMSP  and  model’s  coordinates  was  calculated  as  it  was  for 
the  previous  models.  To  demonstrate  the  difference  in  resolution,  two  examples  of  the 
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SWMF  boundary  plot  are  shown  in  Figure  23.  These  plots  were  generated  with  data 
obtained  on  August  31,  2005  during  low  and  high  Kp  conditions.  Only  one  point 
describes  the  boundary  in  each  MLT  wedge.  This  model  exhibits  minimal  expansion  in 
response  to  geomagnetic  storming  and  a  high  degree  of  symmetry  around  the 
geomagnetic  pole. 


Figure  23.  SWMF  boundary  plots  generated  on  August  31,  2005  for  Kp  3-  (left)  and  Kp  7-  (right) 
conditions.  Boundary  locations  correspond  to  an  energy  flux  of  0.4  erg/cm2/s.  Minimal  expansion  is 
observed  and  a  high  degree  of  symmetry  around  the  geomagnetic  pole  is  observed  with  this  model. 


3.2.5  Assimilative  Mapping  of  Ionospheric  Electrodynamics  Model 

Like  the  NH  and  SWMF  data,  AMIE  data  was  obtained  from  an  external  agency. 
Indebtedness  is  due  to  Dr.  Aaron  Ridley  at  the  University  of  Michigan  for  providing  the 
data  files  containing  energy  flux  information. 

As  was  the  case  with  the  SWMF  model,  AMIE’s  resolution  is  also  1  hour  MLT 
by  0.50°  MLAT.  As  before,  a  MATLAB  program  rounded  the  DMSP  crossing 
coordinates  to  the  nearest  MLT  and  then  found  and  recorded  the  MLAT  corresponding  to 
the  model’s  equatorward  boundary.  The  difference  between  the  two  were  again 
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calculated  and  recorded.  In  order  to  enable  a  graphical  comparison  to  the  other  models, 
another  program  was  developed  to  plot  the  AMIE  equatorward  boundary  on  a  polar 
projection.  An  example  of  the  resulting  plot,  using  a  0.4  erg/cm2/s  threshold,  is  shown  in 
Figure  24. 


Figure  24.  AMIE  boundary  plot  generated  for  May  15,  2005  during  Kp  6-  conditions.  Boundary 
locations  correspond  to  an  energy  flux  of  0.4  erg/cm2/s. 

3.3  Data  Compilation 

Unfortunately,  some  data  availability  limitations  precluded  comparing  all  of  the 
DMSP  data  to  all  of  the  models.  None  of  the  additional  days  selected  for  the  study 
(Table  4)  could  be  processed  with  the  SWMF  or  AMIE  models.  Availability  of  SWMF 
data,  even  for  the  high  Kp  events,  was  further  limited.  Thus,  the  number  of  events 
processed  in  this  statistical  comparisons  of  SWMF  and  AMIE  was  significantly  less  that 
the  total  number  of  DMSP  data  points. 
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Regarding  the  OH  and  OP  models,  many  low-Kp  events  were  also  unable  to  be 
processed.  As  was  discussed  previously,  this  occurred  because  in  some  MLT  sectors 
(particularly  near  1200  MLT),  these  models  did  not  compute  an  energy  flux  above  the 
threshold.  This  problem  was  exacerbated  at  higher  energy  flux  thresholds.  Originally,  an 
additional  flux  threshold  of  1.0  erg/cm2/s  was  also  intended  to  be  used  in  this  study. 
However,  the  losses  associated  with  this  threshold  were  deemed  too  substantial,  and  all  of 
this  data  was  discarded.  Table  7  presents  the  total  number  of  data  points  used  in  the 
analysis  of  each  model. 


Table  7.  Number  of  crossings  for  each  model  for  each  energy  flux  threshold. 


Thresholds 

0.4  0.6  1.0 

Total 

DMSP 

2198 

2154 

1945 

6297 

NH 

1798 

1518 

1007 

4323 

OH 

2059 

1724 

1227 

5010 

OP 

2122 

1722 

1130 

4974 

SWMF 

426 

320 

317 

1063 

AMIE 

849 

764 

610 

2223 

The  data  points  obtained  using  each  specific  threshold  were  not  combined, 
yielding,  in  effect,  three  sub-studies.  At  a  specific  threshold,  all  of  the  data  was  compiled 
into  one  file.  This  file  included  the  UTC,  the  Kp,  the  MLT,  and  the  DMSP  and 
respective  model’s  MLATs.  The  signed  difference  between  the  DMSP  and  each  model 
was  also  included  in  this  file. 

This  file  was  used  to  then  create  histogram  plots  to  convey  the  MLT  distribution 
and  Kp  distribution  of  the  data  points.  The  number  of  data  points  lost  due  to  the  model 
failing  to  produce  a  threshold  crossing  was  also  plotted  as  a  function  of  MLT  and  Kp. 
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In  order  to  afford  the  most  flexibility  to  the  study,  a  scheme  was  devised  to  parse 
the  data  into  the  seasons,  time-of-day  (TOD),  and  Kp  groupings.  Table  8  lists  the 
definitions  of  the  subdivisions  in  each  category. 


Table  8.  Definitions  of  data  subcategories. 


Season 

TOD 

MLT) 

KP 

Winter 

Jan  -  Mar 

Dawn 

04-09 

Low 

0.0 -2.7 

Spring 

Apr  -  Jun 

Day 

10-  15 

Mid 

3.0 -6.0 

Summer 

Jul  -  Sep 

Dusk 

16-21 

High 

6.3  -  9.0 

Fall 

Oct  -  Dec 

Night 

22  -  03 

3.4  Statistical  Measures 

Several  statistical  measures  were  deemed  suitable  for  this  study.  The  first 
calculation  simply  involved  determining  the  mean  (p)  and  standard  deviation  (a)  of  the 
DMSP  data  points  as  a  function  of  Kp.  This  was  used  to  validate  the  statistical  qualities 
of  the  DMSP  data. 

Several  statistics  were  used  to  analyze  the  accuracy  of  the  models.  For  each  of 
the  models,  the  statistical  measures  were  calculated  for  the  all  of  the  available  data  and 
for  various  combinations  of  the  subcategories  listed  in  Table  8. 

The  differences  between  the  DMSP  and  each  model’s  boundary  location  were 
used  to  find  the  mean  value  of  the  differences.  The  standard  deviation  (a)  of  the  mean  of 
these  differences  was  also  computed.  Then,  the  following  formula  was  used  to  calculate 
the  standard  deviation  of  the  differences: 
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where  y  is  the  DMSP  MLAT  and  x  is  the  model’s  ML  AT.  The  standard  deviation 
categorizes  the  spread  of  the  distribution  of  differences  and  is  a  convenient  measure 
because  it  has  units  of  MLAT  and  is  always  positive,  regardless  of  whether  the  model’s 
threshold  is  more  equatorward  or  more  poleward. 

The  next  statistical  measure  that  was  calculated  was  the  ratio  estimate  (RE).  It  is 
given  by 

RE  =  %  (6) 

where  the  bar  operator  designates  the  mean  values.  The  RE  is  a  measure  of  how  closely 
the  model  varies  at  1 : 1  ratio  with  measured  observations.  In  this  study,  model  accuracy 
is  partially  related  to  how  much  the  model’s  equatorward  boundary  moves  in  relation  to 
changes  in  the  DMSP  observation.  If  the  model  and  physical  observations  are  in  perfect 
agreement,  then  RE  —  1.  Any  deviations  from  this  value  indicate  the  model’s  calculated 
boundary  is  varying,  in  an  average  sense,  more  or  less  than  the  DMSP  observed 
boundary. 

One  of  the  most  prolifically  used  statistical  measures  in  a  comparative  study  of 
this  sort  is  prediction  efficiency  (PE).  Prediction  efficiency  is  a  numerical  score  with  a 
maximum  value  of  1  and  no  theoretical  minimum  value.  It  describes  the  percentage  of 
the  observed  variance  (e.g.,  DMSP  data)  explained  by  the  model.  When  the  PE  —  0,  this 
indicates  that  the  model’s  calculations  are  statistically  less  accurate  than  using  the 
observed  mean.  Thus,  negative  values  of  PE  are  not  associated  with  any  discernible 
characteristic  other  than  that  just  described.  The  prediction  efficiency  was  calculated  for 
all  of  the  data  obtained  in  this  study  using  this  fonnula: 
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(7) 


PE  =  1 


Zf=1(yt  -  ^)2 
Ef=1(y*  -  yt)2 


Another  statistic  of  was  used  in  this  study  is  known  as  the  skill  score  ( SS ).  The 
skill  score  compares  a  model’s  performance  against  the  performance  of  a  reference 
model.  It  detennines  if  the  test  model  provides  a  more  accurate  representation  of  the 
observed  conditions  than  does  the  reference.  Statistically,  the  SS  simply  represents  the 
ratio  of  the  square  error  of  two  models.  A  perfect  skill  score  is  1 ,  and  it  also  has  no 
negative  bounds.  It  is  found  by  calculating  the  following: 

If=i (yj  -  xtesti Y 


SS  =  1  - 


EfLi  ( Vi  xref-Sj 


(8) 


A  skill  score  greater  than  0  means  the  tested  model  outperfonned  the  reference 
model  (in  effect,  having  less  aggregate  error).  In  this  study,  the  OH  model  was  selected 
to  be  the  reference  model  because  of  its  long-standing  reputation  and  successful 
implementation  in  various  capacities.  A  SS  was  then  calculated  for  the  NH  and  OP 
models. 
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IV.  Results  and  Analysis 

In  this  chapter,  the  results  and  analysis  of  the  study  will  be  presented.  The 
statistical  characteristics  of  the  DMSP  satellite  data  will  be  discussed  first.  The 
subsequent  section  will  reveal  the  statistical  measures  of  the  models’  independent 
performances.  Then,  the  prediction  efficiency  and  skill  score  comparisons  will  be 
developed.  These  scores  will  be  determined  based  upon  the  data  obtained  using  the 
0.4  erg/cm  /s  energy  flux  threshold  and  will  be  corroborated  with  the  0.6  erg/cm  /s 
dataset.  After  the  prediction  efficiency  rank  order  has  been  established,  the  next  section 
will  present  an  alternative  means  to  compare  the  models’  performances.  Finally,  the 
results  of  a  small-scale,  special  investigation  of  particular  auroral  precipitation  patterns 
will  be  presented. 

As  was  mentioned  in  the  previous  chapter,  the  significant  data  losses  associated 
with  the  1.0  erg/cm  /s  threshold  precluded  using  this  data  in  any  statistical  analysis. 
However,  for  completeness  the  DMSP  statistics  associated  with  the  data  obtained  at  this 
threshold  will  be  included  in  the  discussion.  There  will  also  be  a  small  discussion  on 
why  using  threshold  resulted  in  such  significant  data  losses. 

4.1  DMSP  Data  Characteristics 

Data  from  6,297  DMSP  satellite  passes  were  obtained  in  this  study.  This 
represents  a  sizeable  sample  set  but  is  a  small  fraction  of  the  total  available  DMSP  data. 
The  results  discussed  in  this  section  will  attend  to  the  statistical  characteristics  of  the 
sample  of  DMSP  data.  The  majority  of  the  discussion  in  this  chapter  will  focus  on  data 
obtained  using  a  threshold  flux  of  0.4  erg/cm  /s.  Unless  annotated  specifically,  there  are 
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no  distinct  or  consequential  differences  to  any  conclusions  in  this  chapter  with  regard  to 
the  independent  study  conducted  with  the  0.6  erg/cm  /s  threshold. 

4.1.1  DMSP  Systematics 

Figure  25  contains  the  polar  plots  of  the  DMSP  threshold  crossings  obtained  as  a 
function  of  Kp  index.  Equatorward  expansion  of  the  oval  with  increasing  Kp  value  can 
be  easily  inferred  from  the  plots.  However,  as  was  indicated  in  Figure  18,  the  substantial 
variance  originally  seen  in  the  data  has  persisted. 

The  statistics  of  the  data  are  summarized  in  Table  9,  which  shows  the  mean  value 
(MLAT)  of  the  threshold  crossing  points  (p)  and  the  standard  deviation  (a)  for  each  Kp 
index  value.  Simple  inspection  of  this  data  shows  a  decrease  in  the  average  MLAT  of  the 
effective  boundary  location  as  Kp  index  increases.  This  trend  is  consistent  with  the 
theoretical  expectation  discussed  in  Chapter  II. 


Table  9.  Statistics  of  DMSP  satellite  coordinates  corresponding  to  threshold  crossings.  Average 
crossing  MLAT  is  displayed  for  each  Kp  index  value. 


Threshold  =  0.4  erg/cm2 /sec  (Total Events:  2198) 


Kp 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

N 

21 

235 

381 

404 

562 

307 

184 

41 

63 

0 

H 

74.25 

71.94 

68.53 

66.94 

65.62 

63.74 

62.50 

58.94 

58.60 

0 

a 

2.37 

3.49 

3.86 

3.55 

3.47 

3.52 

4.04 

1.96 

3.35 

0 

Threshold  =  0.6  erg/cm2 /sec  (Total Events:  2154) 


Kp 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

N 

16 

206 

366 

401 

572 

306 

186 

38 

63 

0 

0 

75.37 

72.65 

69.26 

67.75 

66.39 

64.36 

63.00 

59.33 

59.22 

0 

a 

2.55 

3.62 

4.11 

3.78 

3.93 

3.60 

4.04 

2.11 

3.68 

0 

Threshold  =  1.0  erg/cm2 /sec  (Total Events:  1945) 


Kp 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

N 

12 

164 

322 

382 

531 

252 

181 

77 

24 

0 

H 

77.65 

73.63 

69.77 

68.50 

67.57 

65.37 

63.81 

61.17 

56.96 

0 

O 

2.38 

3.70 

4.29 

3.89 

4.06 

4.04 

4.37 

3.35 

2.69 

0 
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Figure  25.  DMSP  boundary  locations  using  a  0.4  erg/cm2/s  energy  flux  threshold  grouped  by  Kp 
index.  Systematic  expansion  equatorward  is  observed  with  increasing  Kp  index,  however,  significant 
variance  exists  in  the  boundary’s  locations. 
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At  the  lowest  energy  flux  threshold,  the  standard  deviations  range  between  1.96 
and  4.04  degrees.  The  standard  deviation  does  not  decrease  in  Kp  bins  where  the  most 
data  was  obtained.  Furthermore,  it  does  not  uniformly  increase  or  decrease  as  a  function 
of  Kp  index  value.  In  effect,  there  is  systemic,  pervasive  scatter  in  the  DMSP  data. 

These  are  qualities  of  the  DMSP  data  that  cannot  be  improved  upon. 

Further  reference  to  Table  9  shows  the  data  obtained  using  the  0.6  and 
1.0  erg/cm  /s  thresholds  exhibit  an  increase  in  standard  deviation  at  almost  all  Kp  levels. 
This  indicates  a  worsening  quality  of  the  data;  however,  the  mean  crossing  values  are 
more  poleward  than  those  at  the  lower  threshold.  This  is  also  theoretically  consistent. 
Thus,  although  the  data  obtained  using  the  lowest  energy  threshold  has  a  sizeable 
variance,  it  remains  the  most  suitable  for  comparison  to  the  models,  and  the  data  obtained 
using  the  larger  threshold  is  suitable  to  support  and  corroborate  any  conclusions  drawn. 

Other  characteristics  of  the  DMSP  data  limit  this  study.  Most  notably,  there  is  a 
non-unifonn  distribution  of  data  points  with  regard  to  MLT.  This  is  the  result  of  the  sun- 
synchronous  orbits  of  the  DMSP  satellites.  The  naturally  infrequent  occurrence  of  events 
with  high  Kp  index  values  also  limits  the  available  data.  The  charts  presented  in  Figure 
26  show  that  there  are  significantly  fewer  crossing  points  obtained  with  a  Kp  index  was 
greater  than  6,  and,  regarding  MLT,  there  exists  a  dearth  of  data  between  11-15  MLT  and 
no  data  between  22-03  MLT.  The  only  way  to  mitigate  either  of  these  limitations  is  to 
increase  the  original  number  of  data  points  by  introducing  data  from  earlier  decades. 
However,  for  the  purposes  of  this  study,  the  amount  of  data  collected  remains  sufficient 
to  draw  valid  and  substantiated  conclusions. 
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Kd  index  Values  MLT 

Figure  26.  DMSP  data  distributions  for  0.4  erg/cm2/s  dataset.  Data  is  limited  at  Kp  index  values 
greater  than  7  and  between  11-15  MLT.  No  data  obtained  between  22-03  MLT. 

It  was  not  apparent  that  other  studies  performed  equivalently  calculated  standard 
deviations,  like  the  values  shown  in  Table  9.  Thus,  it  was  not  possible  to  compare  the 
DMSP  data  obtained  in  this  study  directly  to  that  obtained  in  other  research  efforts. 
However,  because  the  DMSP  data  is  of  central  to  the  validity  of  the  study,  additional 
investigation  was  performed  in  the  form  of  a  basic  regression  analysis.  This  type  of 
analysis  was  performed  in  previous  research  by  Hardy  and  Gussenhoven  and  will  assist 
in  validating  the  characteristics  of  the  DMSP  data  in  this  study. 

4.1.2  Regression  Analysis 

Past  research  studies  (Gussenhoven  et  ah,  1983,  Hardy  et  ah,  2008)  have 
characterized  the  equatorward  boundaries  (A)  obtained  from  DMSP  data  by  conducting  a 
regression  analysis  of  the  form  A  =  A 0  +  a  ■  Kp.  The  slopes  of  the  lines  (a)  can  be 
compared  against  each  other  to  show  sensitivity  to  Kp,  and  the  correlation  coefficients 
(r2)  are  helpful  in  determining  the  linearity  of  the  data.  Data  with  less  scatter  will  have  a 
higher  correlation  coefficient.  The  results  of  regression  analysis  for  the  DMSP  data  in 
this  study  are  contained  in  Table  10. 
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Table  10.  Regression  analysis  of  the  form  A  =  A0  +  a  ■  Kp.  Greyed  areas  represent  MLT  sector 
where  no  data  was  obtained.  Lowest  (worst)  correlation  occurs  in  the  day  sector. 


MLT 

N 

Intercept  (A0) 

Slope  (a) 

I  2, 

|r  | 

1 

-- 

— 

— 

-- 

2 

-- 

— 

— 

-- 

3 

— 

— 

— 

— 

Dawn 

4 

3 

71.75 

-2.81 

1.00 

5 

36 

72.13 

-2.27 

0.74 

6 

116 

71.69 

-1.88 

0.76 

7 

152 

70.91 

-1.56 

0.72 

8 

278 

71.33 

-1.51 

0.70 

9 

200 

72.06 

-1.65 

0.69 

Day 

10 

166 

72.96 

-1.59 

0.63 

11 

94 

71.09 

-0.67 

0.25 

12 

47 

75.27 

-0.98 

0.46 

13 

25 

76.78 

-0.91 

0.70 

14 

44 

75.66 

-0.85 

0.50 

15 

68 

77.06 

-2.00 

0.72 

Dusk 

16 

143 

76.27 

-2.04 

0.76 

17 

180 

74.67 

-2.09 

0.72 

18 

236 

72.60 

-1.88 

0.71 

19 

214 

70.51 

-1.71 

0.75 

20 

145 

69.45 

-1.69 

0.76 

21 

51 

68.84 

-1.75 

0.82 

22 

— 

— 

— 

— 

23 

— 

— 

— 

— 

24 

— 

- 

- 

- 

A  couple  of  salient  features  emerge  as  part  of  the  regression  analysis.  The 
intercepts  between  12-17  MLT  occur  at  higher  latitudes,  which  is  consistent  with 
compression  of  the  magnetopause  on  the  dayside  and  its  associated  magnetic  field  lines. 
Despite  the  smaller  sample  sizes,  the  lower  correlation  coefficients  found  between 
11-14  MLT  (excluding  13  MLT)  is  also  consistent  with  the  fact  the  identification  of  the 
dayside  boundary  is  beset  with  ambiguities,  often  attributed  to  contamination  of  ring 
current  particles  and  influences  from  the  cusp  (Gussenhoven  et  ah,  1981).  There  is  no 
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theoretical  explanation  for  the  increase  in  correlation  at  13  MLT,  and  this  score  is  likely  a 
statistical  anomaly.  In  general,  the  increased  unpredictability  of  the  in  situ  measurements 
inferred  by  the  data  in  this  table  was  also  observed  in  Hardy’s  and  Gussenhoven’s 
previous  research  and  is  supported  theoretically. 

The  reduced  slopes  (a)  between  11-14  MLT  indicate  a  decreased  sensitivity  to 
magnetic  activity  that  has  also  been  corroborated  by  other  studies  of  this  sort.  Hardy  et 
al.  (2008)  observed  a  similar  clustering  of  the  boundaries  in  the  noon  and  mid-afternoon 
MLT  sector  at  Kp  index  values  of  1,  3,  and  5.  Past  studies  have  also  confirmed  the 
existence  of  a  steeper  gradient  in  the  evening  and  dawn  sectors  than  in  the  pre-  and  post- 
noon  sectors  (Gussenhoven  et  al.,  1981).  As  a  result,  the  evening  boundary  is  typically 
less  ambiguous  to  detennine.  Statistically,  this  is  reflected  in  the  higher  correlation 
coefficients  found  outside  the  11-14  MLT  sector. 

With  regard  to  these  morphological  qualities,  the  data  obtained  in  this  study 
closely  aligns  with  these  patterns,  which  is  clearly  demonstrated  by  looking  at  two 
narrow  MLT  sectors  from  opposite  regions.  Figure  27  depicts  two  plots  of  DMSP 
crossing  data  that  were  generated  with  data  occurring  at  1 1  and  20  MLT.  In  the  morning 
sector,  the  equatorward  crossings  are  found  at  more  poleward  locations  than  they  are  at 
20  MLT.  This  is  consistent  with  diurnal  characteristics  of  the  auroral  oval  discussed  in 
Chapter  II.  Furthermore,  the  correlation  coefficient  is  smaller  here,  indicative  of 
increased  uncertainty  in  the  equatorward  boundary’s  position  in  the  pre-  and  post-noon 
sectors,  as  Gussenhoven  et  al.  (1981)  alluded  to.  Finally,  the  slope  of  the  regression  line 
shows  an  increased  sensitivity  to  Kp  in  the  20  MLT  sector  than  in  the  1 1  MLT  sector, 
also  consistent  with  the  greater  expansion  observed  on  the  evening  side  of  the  oval. 
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These  plots  also  clearly  show  the  lack  of  data  for  high-Kp  index  conditions.  This  data 
shortage  occurs  throughout  the  entire  dataset,  regardless  of  sector,  and  precludes  parsing 
the  DMSP  data  into  individual  MLT  sectors. 
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Figure  27.  Scatterplots  of  DMSP  threshold  crossing  coordinates  at  (a)  11  and  (b)  20  MLT.  In  the 
morning  sector,  boundary  is  displaced  poleward,  is  less  dependent  upon  Kp,  and  is  more  scattered. 


4.1.3  Data  Binning 

Table  1 1  depicts  the  results  of  a  regression  analysis  when  this  study’s  data  is 
grouped  into  three  6-hour  MLT  sectors  and  one  18-hour  sector  that  encompasses  all  the 
data.  The  corresponding  data  plots  are  shown  in  Figure  28.  When  compiled  in  this 
manner,  the  data  best  reflects  the  results  found  in  Hardy  et  al.  (2008)  and  Gussenhoven  et 
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al.  (1983),  while  maintaining  the  characteristics  discussed  in  the  previous  paragraphs. 
Importantly,  utilizing  larger  MLT  sectors  also  allows  high-Kp  data  points  to  be  grouped 
together  in  order  to  promote  a  better  statistical  investigation. 


Table  11.  Same  as  Table  10  but  with  MLT  groupings. 


MLT 

N 

Intercept  (A0) 

Slope  (a) 

i  2, 

|r  | 

04-09 

785 

71.50 

-1.64 

0.70 

10-15 

444 

75.13 

-1.69 

0.59 

16-21 

969 

72.60 

-1.94 

0.69 

04-21 

2198 

72.96 

-1.85 

0.66 

Cursory  inspection  of  each  panel  of  Figure  28  reveals  that,  even  when  grouped, 
the  data  still  spans  a  broad  range  of  MLATs  at  each  integral  Kp  value.  In  some  cases, 
individual  threshold  crossings  at  the  same  Kp  index  span  more  than  15°  MLAT.  These 
plots  represent  the  inherent  span  present  in  the  DMSP  data  that  was  accepted  for  the 
calculations  performed  in  this  study.  Additionally,  although  the  data  is  still  scant  at  the 
high-Kp  indexes,  these  groupings  have  populated  this  region  to  the  best  extent  possible. 

4.1.4  Data  Losses 

Because  of  the  sun-synchronous  orbits  of  the  DMSP  satellites,  the  number  of 
crossings  obtained  between  11-13  MLT  was  signifcantly  less  than  in  other  time  frames. 
However,  an  additional  problem  attends  auroral  precipitation  measurement  in  and  near 
the  12  MLT  sector.  Paschmann  (2003)  observed  there  is  a  far  greater  likelihood  of 
measuring  a  flux  above  0.25  erg/cm“/s  between  13.5  and  23  MLT.  Hardy  (2008)  further 
purported  that  lower  rates  of  pitch  angle  scattering  in  the  dayside  plasma  sheet  severly 
restrict  electron  precipitation  in  the  region  centered  on  12  MLT. 
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Figure  28.  DMSP  boundary  coordinates  as  a  function  of  integral  Kp  grouped  (top  to  bottom)  by 
MLT  sectors:  dawn,  day,  dusk,  and  all. 


In  this  study,  however,  166  DMSP  data  points  were  obtained  between  11-13  MLT 
with  an  energy  threshold  of  0.4  erg/cm2/s.  The  OH,  OP,  and  NH  models,  when  run  for 
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the  same  conditions,  often  did  not  determine  a  boundary  in  the  corresponding  MLT 
sector.  In  other  words,  the  model’s  output  never  met  or  exceeded  the  energy  flux 
threshold. 

The  charts  in  Figure  29  show  the  data  loss  statistics  for  each  of  the  models  using  a 
threshold  of  0.4  erg/cnr/s.  The  OP  model  had  3.5%  loss,  found  almost  entirely  between 
1 130-1230  MLT.  The  OH  model’s  loss  percentage  was  6.3%,  and  it  had  the  most  losses 
between  1 100-1300  MLT.  The  NH  model,  however,  did  not  compute  a  boundary  for 
18.2%  of  the  DMSP  data,  and  it  also  exhibited  significant  losses  between  14-16  MLT 
unlike  the  other  models. 

The  loss  patterns  underscore  the  difficulties  associated  with  modeling  the  dayside 
precipitation  patterns.  The  problem  is  only  exacerbated  when  utilizing  higher  flux 
thresholds.  Using  1.0  erg/cnr/s  as  a  threshold,  more  than  35%  of  the  DMSP  data  was 
unusable  because  the  models  did  not  calculate  a  boundary  to  compare  against.  Because 
of  this,  the  0.4  erg/cm  /s  threshold  is  primarily  considered  in  the  detennination  of  the 
models’  perfonnance.  All  of  the  loss  percentages  are  listed  in  Table  12. 


Table  12.  Percentages  of  data  lost  for  the  NH,  OH,  and  OP  models  at  each  energy  flux  threshold. 


Thresholds 

0.4 

0.6 

1.0 

NH 

18.2 

29.5 

48.2 

OH 

6.3 

19.9 

36.9 

OP 

3.5 

17.7 

41.9 
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Figure  29.  Top  row:  Data  losses  using  0.4  erg/cm2/s  threshold  for  NH  model  (left),  OH  model 
(center),  and  OP  model  (right).  Bottom  row:  Percentage  of  available  DMSP  passes  lost  in  each  MLT 
sector. 


There  was  one  additional  factor  to  consider  regarding  the  boundary  produced  by 
the  OP  model.  Even  though  the  OP  model  had  the  lowest  loss  percentage  using  the 
0.4  erg/cm  /s  threshold,  it  did  not  necessarily  calculate  a  physically  reasonable  boundary 
position.  An  example  of  this  characteristic  of  the  OP  model  is  shown  in  Figure  30.  In  the 
14,  15,  and  22  MLT  sectors,  there  are  non-physical  jumps  in  the  equatorward  boundary 
location.  These  boundary  locations  are  particularly  troublesome  because  they  satisfied 
the  more  rigorous  criteria  set  forth  in  Chapter  III  designed  to  reduce  the  OH  model’s 
anomalies.  The  corresponding  synoptic  plot  is  also  shown  in  Figure  30,  which  clearly 
demonstrates  the  scope  of  the  problem.  It  is  apparent  that  requiring  three  consecutive 
energy  flux  values  to  meet  or  exceed  the  threshold  does  eliminate  the  majority  of 
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anomalous  energy  fluxes.  However,  in  order  not  to  introduce  additional  biases  in  this 
comparative  study,  any  boundary  locations  satisfying  the  more  stringent  criterion  were 
included  as  part  of  OP  model’s  accepted  output. 


Figure  30.  OP  model  output  for  data  obtained  on  August  31,  2001  during  Kp  3  conditions.  Data 
points  in  the  14,  15,  and  22  MLT  sectors  demonstrate  peculiar  boundary  locations  that  were 
nevertheless  included  in  the  study  because  they  satisfied  the  already  more  stringent  selection 
requirements. 


4.2  Individual  Model  Characteristics  as  a  Function  of  Kp  Index 

Qualitative  information  about  the  models’  performance  can  be  gained  by  looking  at 
general  patterns  of  their  output.  The  series  of  plots  shown  in  Figure  3 1  depict  the  DMSP 
data  set  overlaid  on  the  same  data  set  for  a  particular  model.  There  is  the  same  number  of 
red  and  blue  points  in  each  plot,  but  due  to  differing  data  losses  among  the  models,  the 
total  number  of  data  points  in  each  plot  is  different.  The  data  is  ordered  such  that  moving 
from  left  to  right  every  subsequent  red  or  blue  point  represents  a  measurement  taken 
during  either  an  equivalent  or  a  higher  Kp  than  the  one  preceding  it.  The  model’s  linear 
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trend  line  is  shown  in  solid  black,  and  the  DMSP  trend  line  is  dashed.  These  lines  depict 
the  best  fit,  but  the  degree  of  correlation  is  not  important  to  the  ensuing  qualitative 
discussion. 

Although  the  DMSP  and  model  data  exhibit  significant  scatter,  the  results 
obtained  are  consistent  with  theoretical  prediction.  The  trend  lines  clearly  show  that  each 
model  exhibits  an  equatorward  migration  of  its  boundary  crossing  MLAT  coordinates 
with  increasing  Kp.  By  comparing  the  trend  lines,  the  OH  model  appears  to  qualitatively 
match  the  DMSP  data  the  best,  but  some  of  the  other  models  also  show  good 
performance.  The  OP  model’s  trend  line  more  closely  matches  the  DMSP  data  as  Kp 
increases,  whereas,  the  NH  and  SWMF  models  go  from  overestimating  the  boundary  at 
low  Kps  to  slightly  underestimating  it  at  high  Kps.  Finally,  the  AMIE  model  depicts  a 
nearly  uniform  equatorward  bias  at  all  Kps,  but  the  slopes  of  the  trend  lines  appear  nearly 
equal. 

Thus,  the  series  of  plots  in  Figure  3 1  provide  the  first  qualitative  indication  that  a 
model’s  accuracy  may  be  dependent  upon  Kp  index.  The  SWMF  model  is  a  particularly 
good  example  of  this.  This  model’s  performance  appears  to  be  very  poor  at  low  Kps,  but 
as  the  Kp  index  increases,  the  deviation  becomes  much  smaller. 
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Figure  31.  Plots  of  DMSP  and  NH,  OH,  OP,  SWMF,  and  AMIE  (top  to  bottom)  as  functions  of 
increasing  Kp  between  Kp  1  and  Kp  8+.  Black  lines  indicate  data  trend. 


4.3  Model  Accuracy  Comparisons 

To  begin  assessing  and  comparing  the  performance  of  the  models,  the  broadly 
applicable  ratio  estimate  will  be  used  to  expand  upon  some  of  the  observed  qualitative 
trends.  Then,  the  remainder  of  this  section  will  present  the  results  of  the  comprehensive 
statistical  study  designed  to  measure  each  model’s  accuracy.  This  will  include  a 
discussion  of  average  deviations  and  prediction  estimate  scores.  Finally,  the  most 
successful  models  will  be  compared  to  each  other,  and  through  the  use  of  the  skill  scores, 
a  most  accurate  model  will  be  designated. 
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4.3.1  Ratio  Estimate 


The  ratio  estimate  (RE)  was  defined  in  Chapter  III  and  pertains  to  the  data  and 
applies  to  the  average  characteristics  of  the  dataset  in  question.  The  optimum  RE  value  is 
1,  indicating  the  means  of  the  two  data  sets  are  equal.  If  the  RE  is  greater  than  1,  then  the 
model’s  average  output  is  more  equatorward  that  the  DMSP  satellite  data.  An  RE  value 
less  than  1  indicates  the  opposite. 

The  RE  can  be  misleading  because  small  deviations  may  be  indicative  of  much 
larger  errors.  For  example,  an  RE  of  1.05  at  70°  MLAT  corresponds  to  a  3.5°  deviation, 
which  is  equivalent  to  a  transverse  distance  of  approximately  430  km  at  a  DMSP 
satellite’s  orbit.  Each  model’s  ratio  estimates  are  displayed  in  Table  13. 

A  few  trends  are  observed  with  regard  to  RE.  First,  SWMF  and  AMIE  have  high 
RE  values  in  the  low  and  moderate  Kp  index  ranges  but  perform  better  during  stonny 
conditions.  All  of  the  models  perform  well  during  high-Kp  events,  except  for  the  NH 
model,  which  underestimates  the  equatorward  expansion  of  the  oval.  Otherwise,  both  the 
NH  and  OH  models  have  RE  values  very  close  to  1.00  during  all  conditions.  The  OP 
model  also  performs  suitably  well,  though  at  low  and  moderate  Kps,  it  tends  to  locate  the 
boundary  more  equatorward  than  is  seen  in  the  DMSP  data.  However,  because  the  ratio 
estimate  is  solely  based  upon  mean  values,  it  only  describes  the  general  correlation 
between  the  model  and  DMSP  data  without  regard  for  any  variance  in  the  data.  With  this 
in  mind,  the  study  now  looks  to  complement  the  RE  statistics  by  considering  the  standard 
deviation  of  each  model’s  data  set  as  well. 
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Table  13.  Ratio  estimates  for  each  model  binned  by  Kp  conditions. 


NH 

OH 

OP 

SWMF 

AIME 

All  Kp 

1.00 

1.00 

1.02 

1.06 

1.05 

LowKp 

1.01 

1.00 

1.03 

1.11 

1.06 

Mod  Kp 

1.01 

1.01 

1.02 

1.06 

1.06 

High  Kp 

0.96 

0.98 

0.99 

0.99 

1.01 

4.3.2  Mean  Deviation  between  DMSP  and  Model 

Figure  32  presents  five  charts  that  summarize  the  basic  statistics  of  each  model’s 
deviation  from  the  DMSP  data.  These  charts  were  constructed  using  all  of  the  data 
obtained  in  this  study  and  do  not  represent  any  subdivision  into  any  of  the  previously 
described  categories.  These  charts  summarize  the  results  of  calculating  the  individual 
differences  between  each  DMSP  data  point  and  the  corresponding  data  point  detennined 
by  each  of  the  models.  If  a  difference  is  negative,  it  indicates  the  model’s  boundary  is 
more  poleward  (i.e.,  at  a  higher  MLAT)  than  the  DMSP  boundary.  A  perfect  model 
would  have  a  mean  difference  of  0,  and  the  standard  deviation  of  the  distribution  plot  of 
the  difference  statistics  would  also  be  equal  to  0.  Therefore,  small  values  in  both 
statistics  are  indicative  of  accurate  model  performance. 

The  plots  in  Figure  32  depict  the  distribution  of  all  of  the  differences  rounded  to 
the  nearest  integer  between  -10  and  10.  If  the  difference  between  the  DMSP  boundary 
and  a  model’s  boundary  exceeded  ±10,  it  was  counted  in  the  10  or  -10  bins  respectively. 
To  facilitate  comparison,  the  mean  value  of  the  differences  and  the  standard  deviation  of 
the  differences  were  calculated. 
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DMSP-AMIE  MLAT  (degrees) 

Figure  32.  MLAT  deviations  (DMSP-model)  binned  in  integral  steps  using  0.4  erg/cm2/s  threshold. 
Entire  data  set  has  been  included.  NH  (green),  OH  (yellow),  OP  (red),  SWMF  (blue),  AMIE  (cyan). 


The  NH  model’s  average  deviation  was  0.20°  MLAT,  and  the  OH  model’s  was 
0.16°.  The  OP  model’s  average  displacement  was  1.16°,  indicating  it  calculated  a  more 
equatorward  boundary  on  average.  This  result  is  consistent  with  the  qualitative  trends 
observed  in  Figure  31.  The  standard  deviation  of  the  NH,  OH,  and  OP  models  is  3.05, 
3.12,  and  2.94°  respectively.  As  a  reminder,  these  values  represent  standard  deviations  of 
the  deviation  data.  No  other  similar  study  published  the  standard  deviations  in  this  way, 
so  it  is  impossible  to  compare  these  results  to  others.  However,  an  error  of  ±3°  MLAT 
equates  to  more  than  350  km  of  transit  for  a  DMSP  satellite  orbit,  so  the  amount  of 
uncertainty,  as  revealed  by  these  statistics  is  considerable. 
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The  similarity  in  standard  deviations  for  NH,  OH,  and  OP  models  is  interesting, 
though,  because  the  OH  and  NH  models  are  highly  discretized  (binned  by  integral  Kp 
value),  whereas  the  OP  model  ingests  many  dynamic  parameters  tied  to  small  changes  in 
the  geomagnetic  conditions.  Said  differently,  the  considerably  more  computationally 
complex  OP  model  does  not  achieve  noticeably  lower  deviations  from  the  DMSP  data 
than  does  its  more  simply  designed  peers. 

The  average  perfonnance  of  SWMF  and  AMIE  are  poor  by  comparison.  Not  only 
did  both  models  locate  the  boundary  at  significantly  lower  MLATs,  but  both  also  have 
high  standard  deviations.  Because  these  calculations  are  based  upon  the  entire  data  set, 
there  is  no  consideration  given  to  the  possibility  of  a  model  exhibiting  a  high  degree  of 
accuracy  in  a  particular  subcategory.  If  the  data  is  broken  down  into  Kp  groups,  as  was 
done  with  the  RE  calculation,  the  performance  statistics  change. 

For  example,  when  only  high-Kp  conditions  (i.e.,  Kp  >  6-)  are  considered,  SWMF 
and  AMIE  experience  marked  improvement.  All  of  the  models’  perfonnance  results 
under  these  conditions  are  shown  in  Figure  33.  Under  these  circumstances,  the  mean 
deviations  of  NH,  OH,  and  OP  are  now  negative,  meaning  these  model’s  boundary 
locations  are  more  poleward  on  average.  The  NH  model’s  mean  deviation  shifted  2.96°, 
and  the  OH  and  OP  model  shifted  1.14°  and  1.94°  respectively.  These  shifts  represent  a 
significant  change  in  each  model’s  performance. 

The  SWMF  model’s  mean  deviation  experiences  a  3.03°  change,  going  from 
3.39°  to  -0.36°.  This  is  the  likely  result  of  the  model’s  effective  indifference  to  Kp 
conditions.  Put  a  different  way,  the  SWMF  model  appears  to  be  preconditioned  to  model 
high-Kp  events  only,  which  is  why  its  scores  are  poorer  during  quieter  conditions.  The 
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AMIE  model  appears  to  improve,  but  the  standard  deviation  of  its  data  warrants  caution 


in  making  any  conclusions.  Its  deviations  are  more  spread  out  than  the  other  models, 


which  definitely  limits  any  confidence  in  its  reliability. 


DMSP  AMIE  MLAT  (degrees) 


Figure  33.  Same  as  Figure  32  but  for  Kp  >  6-  only. 


Categorizing  the  models  based  upon  geomagnetic  activity  levels  is  one  way  the 
models  can  be  measured.  However,  based  upon  the  earlier  discussion  of  the  variability  in 
the  aurora  based  upon  MLT  sector,  this  study  also  sought  to  examine  any  changes  to  the 
models’  accuracies  with  regard  to  time  of  day  and  time  of  year.  As  the  data  is  grouped 
into  smaller  and  more  specific  representations,  some  models  exhibit  particular  strengths. 
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Table  14  lists  the  models  with  the  lowest  average  deviation  in  each  subcategory.  Spring 
and  fall  seasonal  statistics  were  excluded  because  not  enough  data  during  these  seasons 
was  introduced  into  this  study.  The  same  circumstance  was  true  for  the  10-15  MLT 
statistics  during  high  Kp  conditions.  These  unfortunate  gaps  represent  an  avenue  for 
continued  research. 


Table  14.  Lowest  average  deviations  during  dawn,  day,  and  dusk  MLT  sectors  and  winter  and 
summer  seasons  for  high,  moderate,  and  low  Kp  conditions.  Table  entries  represent  deviation  in 
degrees  of  MLAT.  Direction  of  deviation  (equatorward  or  poleward)  was  not  factored  into  ranking. 


MLT 

High  Kp 

Moderate  Kp 

LowKp 

04-09 

SWMF 

-0.33 

NH 

0.18 

OH 

0.38 

10-15 

-- 

-- 

NH 

0.40 

NH 

-0.40 

16-21 

SWMF 

-0.67 

OP 

-0.87 

OH 

-0.32 

Season 

Hig 

ti  Kp 

Moderate  Kp 

LowKp 

Winter 

SWMF 

0.17 

NH 

0.67 

OH 

0.59 

Summer 

OH 

-0.07 

NH 

0.13 

NH 

0.31 

During  storming  events,  the  SWMF  model’s  boundary  location  is  the  closest  on 
average  regardless  of  the  MLT  sector,  and  it  also  models  well  during  winter.  During 
moderate -Kp  events,  the  NH  model  performs  best  from  04-15  MLT,  and  the  OP  model 
does  the  best  from  16-21  MLT.  The  NH  model  remains  the  top  performer  during  winter 
and  summer.  During  quiet  conditions,  the  equatorward  boundary  is  most  closely 
represented,  on  average,  by  the  OH  model  from  04-09,  from  16-21  MLT,  and  during 
summer.  The  NH  model  has  the  lowest  deviation  from  10-15  MLT  and  during  summer. 

Although  none  of  the  models  is  clearly  dominant  in  all  categories,  based  upon 
these  results,  the  NH,  OH,  OP,  and  SMWF  models  have  earned  credibility  in  specific 
niches.  Each  of  these  models  demonstrates  comparatively  good  performance  in  at  least 
one  Kp  and/or  MLT  regime.  The  SWMF  model  is  ranked  highest  in  2  of  8  MLT 
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categories  and  1  seasonal  category.  The  OH  model  earns  2  MLT  categories  and  2 
seasonal  categories.  The  NH  model  has  the  highest  number  of  categories — 3  MLT  and  3 
seasonal.  Somewhat  surprisingly,  the  OP  model  only  has  the  lowest  deviation  in  one  of 
the  MLT  categories. 

4.3.3  Prediction  Efficiency  (PE) 

The  mean  deviation  is  an  important  consideration,  but  it  does  not  tell  the  full 
story.  Comparative  studies  of  this  sort  are  well-suited  for  prediction  efficiency  (PE) 
calculations.  The  PE  is  useful  because  it  measures  the  percentage  of  variance  in  the 
observed  data  (here,  DMSP  data)  that  is  explained  by  the  model.  Thus,  a  perfect  PE 
score  is  1,  indicating  100%  of  the  model’s  variance  has  been  captured  by  the  model. 
Contrarily,  any  score  less  than  0  indicates  the  model’s  variance  exceeds  the  variance  in 
the  DMSP  data.  In  this  unfavorable  case,  using  the  mean  of  the  observed  data  serves  as  a 
more  accurate  predictor,  on  average,  than  using  the  model. 

Table  15  lists  the  PE  scores  when  the  data  is  divided  into  Kp  groups  and  then  is 
further  subdivided  into  MLT  sectors.  In  order  for  a  score  to  be  calculated,  more  than  20 
data  points  had  to  exist  within  the  category.  The  top  line  of  the  table  lists  the  PE  scores 
for  the  entire  data  set.  In  this  aggregate  condition,  the  OP  model  receives  the  highest  PE 
score.  It  is  followed  by  the  OH  model  and  then  the  NH  model.  Neither  the  SWMF  nor 
AMIE  model  has  a  positive  PE  when  all  of  the  data  is  considered. 
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Table  15.  Prediction  efficiency  scores.  Negative  PEs  are  indicated  by  (--).  Green  box  represents  best 
PE  in  that  category. 


MLT 

Kp 

NH 

OH 

OP 

SWMF 

AMIE 

04-21 

All 

0.45 

0.51 

0.55 

— 

— 

04-21 

High 

— 

— 

0.13 

0.29 

— 

Mod 

0.32 

0.31 

0.30 

-- 

-- 

Low 

0.22 

0.34 

0.37 

— 

— 

04-09 

High 

— 

— 

— 

0.32 

— 

Mod 

0.19 

0.05 

0.30 

-- 

-- 

Low 

0.19 

0.06 

0.10 

— 

— 

10-15 

High 

Insufficient  Data 

Mod 

0.06 

0.21 

0.12 

-- 

-- 

Low 

— 

0.40 

0.38 

— 

— 

16-21 

High 

— 

— 

0.15 

0.25 

— 

Mod 

0.39 

0.35 

0.19 

— 

-- 

Low 

0.07 

0.36 

0.37 

— 

— 

The  OP  model  receives  a  positive  PE  in  all  of  the  categories  except  for  high-Kp 
events  occurring  between  04-09  MLT,  which  indicates  strong  perfonnance.  However,  its 
scores  during  the  other  high-Kp  categories  are  relatively  low.  The  OH  and  NH  models 
fare  even  worse  during  periods  of  significant  geomagnetic  storming;  both  have  negative 
scores  in  each  of  the  MLT  categories  during  high-Kp  conditions.  The  SWMF  model  has 
the  highest  PE  scores  in  all  MLT  sectors  during  high-Kp  events.  This  fact,  combined 
with  its  low  mean  deviation,  indicate  the  SWMF  model  is  the  most  accurate  during  high- 
Kp  events. 

There  is  no  decisive  winner  when  considering  the  PEs  in  Table  15  during  other 
activity  levels  and  times  of  day,  and  any  broad  conclusions  would  be  rife  with  caveats. 
For  example,  when  the  Kp  index  is  low,  both  OP  and  OH  fare  well,  but  not  in  the  dawn 
sector,  where  the  NH  model  has  the  highest  PE.  However,  the  top-ranked  NH  model 
only  accounts  for  19%  of  the  variance  of  the  DMSP  data  in  this  subcategory,  and  this 
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underscores  poor  performance  for  all  of  the  models.  A  similar  situation  occurs  during 
moderate -Kp  events  between  10-15  MLT.  In  this  case,  the  OH  model  only  accounts  for 
21%  of  the  DMSP  variance,  but  this  score  is  almost  twice  the  OP  model’s  score  of  12%. 

The  PEs  calculated  using  data  occurring  during  winter  and  summer  are  listed  in 
Table  16.  Generally  speaking,  the  PE  scores  are  slightly  higher  than  what  was  found  in 
the  previous  table.  Except  for  case  of  high  Kp  during  winter,  the  OP  model  performs 
well  in  all  categories  and  could  justifiably  be  declared  the  best  performing  model  in 
consideration  of  these  categories.  Combined  with  the  previous  results,  the  OP  model  has 
begun  to  establish  itself  as  a  front-runner. 

But,  other  models  also  perfonn  well  under  certain  circumstances.  The  OH  model 
does  comparatively  well  during  the  summer  months  and  also  during  the  moderate-Kp 
events  during  the  winter.  It  also  does  quite  well  in  the  summer  months  when  the  Kp 
index  is  high.  The  NH  model  does  not  have  any  of  the  highest  scores,  but  it  does  not 
necessarily  languish  either.  It  performs  poorly  during  high  and  low-Kp  events  in  the 
winter  and  high-Kp  events  in  the  summer.  The  SWMF  model  again  performs  well  in  a 
high-Kp  category,  as  it  has  the  only  positive  PE  score  during  the  winter  months. 


Table  16.  Same  as  Table  15  but  data  is  grouped  by  season. 


Season 

Kp 

NH 

OH 

OP 

SWMF 

AMIE 

Winte  r 

High 

-- 

-- 

-- 

0.30 

-- 

Mod 

0.38 

0.46 

0.45 

-- 

-- 

Low 

— 

0.03 

0.29 

— 

— 

Summer 

High 

— 

0.45 

0.28 

0.08 

— 

Mod 

0.30 

0.28 

0.49 

— 

-- 

Low 

0.28 

0.34 

0.46 

— 

— 
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4.3.4  Skill  Score 


The  last  performance  measure  investigated  in  this  study  is  the  skill  score  (SS). 
Development  of  meaningful  skill  scores  requires  the  selection  of  one  model  as  the 
reference.  For  the  purposes  of  this  study,  the  OH  model  will  be  used  in  this  capacity,  and 
it  will  be  compared  to  the  NH  and  OP  models.  The  SWMF  model’s  perfonnance  in  all 
other  categories  except  for  high-Kp  and  the  AMIE  model’s  overall  performance  do  not 
warrant  performing  a  skill  score  analysis  on  these  models. 

A  perfect  SS  is  1,  which  indicates  the  test  model  and  all  observations  are  in 
perfect  agreement.  A  positive  SS  means  the  test  model  provides  a  better  forecast  than  the 
reference  model,  but  the  SS  does  not  necessarily  imply  any  degree  of  accuracy.  That  is, 
if  the  reference,  itself,  was  inaccurate,  the  SS  only  indicates  whether  or  not  the  test  model 
performed  better.  However,  there  is  a  distinct  possibility  the  test  model’s  predictive 
capacity  is  also  quite  poor. 

The  data  in  Table  17  summarizes  the  SS  results.  A  green  box  designates  a 
positive  score.  Analysis  of  the  skill  scores  shows  that  the  NH  model  is  an  improvement 
over  the  OH  model  under  the  majority  of  low-  and  moderate -Kp  events.  However,  it 
must  be  noted  that  all  of  the  NH  scores  are  only  slightly  positive  and  are  less  than  0. 15. 
This  indicates  a  very  modest  improvement  over  the  OH  model.  By  contrast,  the  OP 
model  has  fewer  positive  scores,  but  some  of  its  scores  are  much  greater  than  the  NH 
model’s.  The  OP  model  provides  a  particular  advantage  over  the  OH  model  during  high 
Kp  events. 

These  SS  results  are  intended  to  complement  the  PE  analysis,  but  nuances  do 
exist.  In  the  case  of  the  low-Kp  events  occurring  in  the  16-21  MLT  sector,  the  results 
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appear  to  be  in  stark  contrast  to  those  found  using  the  PE.  The  number  of  NH  and  OP 
data  points  used  in  computing  the  skill  score  was  not  necessarily  the  same,  due  to  the 
potential  for  data  losses  described  in  an  earlier  section.  In  this  case  highlighted  here,  the 
OP  skill  score  was  determined  using  235  points,  while  the  NH  skill  score  only  used  124 
points.  Refer  to  Figure  29  for  the  specific  loss  statistics  associated  with  the  16-21  MLT 
sector. 


Table  17.  Skill  scores.  Green  boxes  indicate  the  model  performed  better  than  the  OH  reference 
model. 


MLT 

Kp 

n 

NH 

n 

OP 

04-21 

All 

1763 

0.07 

2018 

0.01 

High 

133 

-0.37 

140 

0.37 

04-21 

Mod 

1198 

0.14 

1299 

-0.02 

Low 

432 

0.05 

579 

-0.05 

High 

58 

-0.53 

58 

0.30 

04-09 

Mod 

486 

0.15 

486 

0.27 

Low 

231 

0.13 

231 

0.05 

High 

Insuffici 

ent  D; 

ita 

10-15 

Mod 

146 

0.14 

198 

-0.06 

Low 

77 

-0.40 

113 

-0.34 

High 

63 

-0.60 

67 

0.18 

16-21 

Mod 

566 

0.13 

615 

-0.24 

Low 

124 

0.09 

235 

-0.07 

Again,  like  with  the  PE,  the  SS  can  be  also  determined  with  a  seasonal 
dependence.  Table  18  summarizes  those  results.  The  results  are  similar  to  the  previous 
findings.  Again,  the  OP  model  demonstrates  a  greater  improvement  over  the  reference 
than  does  the  NH  model.  Neither  model  performs  better  than  OH  during  high-Kp 
summer  events. 
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Table  18.  Same  as  Table  17  but  with  data  grouped  by  season. 


Season 

Kp 

n 

NH 

n 

OP 

Winter 

High 

49 

-0.39 

55 

0.40  : 

Mod 

275 

0.14 

300 

0.04 

Low 

68 

0.01 

95 

-0.05 

Summer 

High 

44 

-1.18 

44 

-0.03 

Mod 

260 

0.07 

277 

0.31 

Low 

211 

0.02 

288 

0.20 

4.4  Comparative  Statistics  Using  Different  Thresholds 

As  was  mentioned  in  the  first  section  of  this  chapter,  the  same  experimental 
procedures  were  performed  when  representing  the  boundary  with  higher  energy  flux 
thresholds.  The  original  intention  of  this  was  to  vet  the  results  found  using  the 
0.4  erg/cm  /  threshold  with  results  obtained  using  0.6  and  1.0  erg/cm  /s  thresholds.  It 
was  discovered  the  model’s  often  did  not  compute  any  flux  values  at  or  above 
1 .0  erg/cm  /s,  reducing  the  available  data  set  by  nearly  50%. 

In  some  cases,  especially  during  low-Kp  events,  the  DMSP  satellite  also  did  not 
record  such  a  high  energy  flux  value.  However,  the  more  prolific  problem  with  the 
DMSP  data  pertained  to  the  physical  processes  occurring  in  the  magnetosphere.  In 
Chapter  II,  it  was  shown  the  energy  dependent  dominance  of  the  gradient  drift  results  in  a 
steep  energy  gradient  that  is  often  observed  at  the  equatorward  boundary  of  the  auroral 
oval.  However,  particularly  during  quiet  activity,  particle  energies  in  the  boundary 
region  of  the  plasma  sheet  are  reduced,  and  this  causes  a  shallower  onset  of  energies. 
Large  energy  fluxes,  however,  may  still  exist  well  inside  of  the  auroral  oval  due  to  the 
diverse  population  of  particle  energies  found  in  central  plasma  sheet. 
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Both  of  these  circumstances  are  depicted  in  Figure  34,  which  depicts  two  satellite 
passes  during  low  Kp  conditions.  In  the  panel  on  the  left,  the  satellite  did  not  encounter 
an  energy  flux  value  above  the  1.0  erg/c nr/s  threshold,  where  it  would  have  for  a  lower 
fixed  value.  In  the  plot  on  the  right,  the  threshold  is  crossed  well  inside  of  the  auroral 
oval,  which  is  clearly  not  indicative  of  the  equatorward  boundary.  As  a  result,  it  was 
determined  that  this  flux  threshold  could  not  be  reliably  and  consistently  associated  with 
the  equatorward  boundary,  and  the  data  was  removed  from  the  study. 

However,  the  data  obtained  using  a  0.6  erg/cm  /s  threshold  was  set  an  appropriate 
level  and  did  not  encounter  the  same  issues  with  data  losses  and  unrepresentative 
threshold  crossings.  Thus,  the  data  obtained  with  this  threshold  was  used  to  correlate  to 
the  results  obtained  using  0.4  erg/cm  /s.  The  distribution  plots  of  the  differences  are 
shown  in  Figure  35.  Again,  this  data  was  calculated  using  the  entire  data  set  and  was  not 
divided  into  any  subcategories. 
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Figure  34.  Examples  of  data  loss  using  1.0  erg/cm2/s  threshold.  Top  plot  demonstrates  situation 
where  no  boundary  occurred  due  to  low  energy  throughout  the  auroral  zone.  Bottom  plot 
demonstrates  a  misplaced  boundary  (black  arrow)  located  well  inside  auroral  oval  based  upon 
central  plasma  sheet  particle  energies. 
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DMSP-AMIE  MLAT  (degrees) 


Figure  35.  MLAT  deviations  (DMSP-model)  binned  in  integral  steps  using  0.6  erg/cm2/s  threshold. 
Entire  data  set  has  been  included.  NH  (green),  OH  (yellow),  OP  (red),  SWMF  (blue),  AMIE  (cyan). 


These  charts  show  similar  results  to  what  was  calculated  using  the  lower 
threshold.  The  OH  and  NH  models  have  the  lowest  average  deviations.  The  OP  model 
exhibits  an  improvement  in  its  mean  difference  and  has  a  standard  deviation  that  is  very 
low  compared  to  its  peers.  Neither  the  SWMF  nor  the  AMIE  models  exhibit  an 
enhancement  in  accuracy.  Table  19  provides  a  comparison  of  each  model’s  difference 
statistics  using  both  thresholds. 
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Table  19.  Comparison  of  mean  deviation  and  standard  deviation  using  different  energy  flux 
thresholds. 


Thresh 

NH 

OH 

OP 

SWMF 

AMIE 

0.4 

N 

1798 

2059 

2122 

426 

849 

P 

0.20 

0.16 

1.16 

3.39 

3.29 

o 

3.05 

3.12 

2.94 

4.21 

4.49 

0.6 

N 

1518 

1724 

1772 

320 

764 

P 

0.15 

-0.01 

0.76 

3.60 

2.90 

o 

3.14 

3.31 

2.79 

4.38 

4.30 

The  data  in  Table  20  summarizes  the  prediction  efficiency  scores  when  using  a 
flux  threshold  of  0.6  erg/cmVs.  As  was  found  with  the  lower  threshold,  the  SWMF 
model  has  the  highest  PE  during  high-Kp  conditions  from  04-15  MLT,  but  at  the  higher 
threshold,  the  OP  model  now  has  the  highest  score  in  evening  sector.  The  OP  model 
achieves  higher  scores  in  the  majority  of  the  other  categories. 


Table  20.  Prediction  efficiency  score  comparison  using  0.6  erg/cmVs  threshold.  Negative  PEs  are 
indicated  by  (--).  Green  box  represents  highest  PE  in  that  category. 


MLT 

Kp 

NH 

OH 

OP 

SWMF 

AMIE 

04-21 

All 

0.44 

0.41 

0.58 

- 

- 

04-21 

High 

— 

— 

0.16 

0.24 

— 

Mod 

0.31 

0.29 

0.39 

— 

— 

Low 

0.14 

0.11 

0.47 

- 

- 

04-09 

High 

— 

-- 

— 

0.55 

— 

Mod 

0.16 

0.07 

0.31 

— 

— 

Low 

0.11 

— 

0.20 

- 

- 

10-15 

High 

Insufficient  Data 

Mod 

— 

0.12 

0.40 

— 

0.21 

Low 

— 

0.17 

0.58 

- 

- 

16-21 

High 

— 

— 

0.29 

0.16 

— 

Mod 

0.37 

0.40 

0.32 

— 

— 

Low 

— 

0.13 

0.40 

- 

- 
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Comparing  the  PE  results  for  each  threshold  reveals  that  10  of  10  OP  model 
scores  were  higher  using  the  higher  energy  flux  threshold,  while  5  of  5  NH  scores  and  5 
of  7  OH  scores  were  lower.  The  reason  for  the  surge  in  OP  performance  is  not  inherently 
obvious  but  can  be  partially  explained  be  investigating  differences  between  the  data  sets 
obtained  using  the  different  thresholds. 

Referring  back  to  Table  9,  the  DMSP  data  obtained  using  the  0.6  erg/cm  /s 
threshold  has  a  larger  variance  than  the  data  obtained  using  the  lower  threshold  in  8  of  the 
9  Kp  categories.  Referring  to  Equation  (7),  if  the  differences  between  the  model  and 
observed  data  remain  the  same,  a  larger  variance  in  the  DMSP  data  should  result  in 
higher  PE  scores  for  all  of  the  models.  Thus,  in  order  for  the  NH  and  OH  model’s  scores 
to  have  decreased,  the  average  difference  between  each  of  them  and  the  DMSP  data  must 
have  increased.  This  is  the  tenn  in  the  numerator  of  Equation  (7). 

In  Figure  36,  the  corresponding  threshold  crossing  points  for  the  low  (blue)  and 
high  (red)  flux  thresholds  are  overlaid  on  the  same  plot  for  the  NH,  OH,  and  OP  models. 
The  boundaries  in  these  plots  correspond  to  a  Kp  index  of  3.  Panel  (a)  and  panel  (b) 
show  the  NH  and  OH  model  boundaries  are  virtually  identical.  Although  the  higher  flux 
boundary  is  very  slightly  displaced  toward  the  pole,  which  is  consistent  with  theory,  there 
is  very  little  displacement.  Panel  (c)  shows  the  OP  model’s  output,  and  its  higher  energy 
boundary  is  also  more  poleward  as  well.  However,  its  displacement  is  slightly  larger. 

Again,  referring  to  the  statistics  in  Table  9,  the  DMSP  crossing  points  are  located, 
on  average,  more  poleward  at  the  higher  flux  threshold.  This  more  significant 
displacement  of  the  OP  model  thus  reduces  the  term  in  the  numerator  of  Equation  (7) 
more  so  than  either  the  NH  or  OH  models.  This,  results  in  higher  PE  scores  for  the  OP 


89 


model.  The  NH  and  OH  models,  whose  outputs  are  virtually  indistinguishable,  are  better 
suited  modeling  boundaries  associated  with  lower  energy  fluxes. 


Figure  36.  Boundary  plots  at  0.4  (red)  and  0.6  (blue)  erg/cm2/s  thresholds  for  (a)  NH,  (b)  OH,  and 
(c)  OP.  Plots  represent  conditions  with  Kp  3. 

A  list  of  the  model  with  the  top  prediction  efficiency  score  in  each  subcategory 
for  both  flux  thresholds  is  provided  in  Table  21.  Some  general  conclusions  can  be 
gleaned  from  the  low-flux  results.  The  SWMF  model  has  clearly  carved  out  a  niche 
modeling  the  oval’s  equatorward  boundary  during  most  high-Kp  events.  The  OP  model 
does  well  at  modeling  many  of  the  low-Kp  conditions.  Moderate-Kp  events,  those  with 
Kp  index  values  between  3  and  6,  are  relatively  split  between  the  NH,  OH,  and  OP 
models.  If  the  data  is  parsed  into  any  subcategorizes  at  all,  then  the  OP  model  has  the 
highest  PE  scores  using  both  thresholds. 
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Table  21.  Top  prediction  efficiency  scores  in  each  subeategory  using  0.4  and  0.6  erg/cm2/s  thresholds. 


0.4 

0.6 

SEASONS 

Winter 

High 

SWMF 

0.30 

— 

— 

Mod 

OH 

0.46 

OP 

0.51 

Low 

OP 

0.29 

OP 

0.16 

D£ 

S 
•— 1 

Sm 

cl 

High 

SWMF 

0.22 

SWMF 

0.21 

Mod 

NH 

0.14 

OP 

0.16 

C ft 

Low 

OP 

0.32 

OP 

0.15 

Summer 

High 

OH 

0.45 

OP 

0.32 

Mod 

OP 

0.49 

NH 

0.33 

Low 

OP 

0.46 

OP 

0.60 

Fall 

High 

— 

— 

— 

— 

Mod 

NH 

0.38 

OP 

0.52 

Low 

OH 

0.44 

OH 

0.45 

MLT SECTORS 

04-09 

High 

SWMF 

0.32 

SWMF 

0.55 

Mod 

OP 

0.30 

OP 

0.31 

Low 

NH 

0.19 

OP 

0.20 

10-15 

High 

-- 

-- 

-- 

— 

Mod 

OH 

0.21 

OP 

0.40 

Low 

OH 

0.40 

OP 

0.58 

16-21 

High 

SWMF 

0.25 

OP 

0.29 

Mod 

NH 

0.39 

OH 

0.40 

Low 

OP 

0.37 

OP 

0.40 

ALL 

ALL 

OP 

0.55 

OP 

0.58 

An  attempt  to  corroborate  these  broad  conclusions  using  a  higher  energy  flux 
results  highly  favors  to  the  OP  model,  which  earned  the  top  score  in  13  of  18  categories 
in  which  data  was  available.  Table  22  lists  the  models  that  had  the  highest  PE  score 
using  both  the  low-  and  high-energy  flux  data.  This  circumstance  occurred  8  times  out  of 
18  opportunities.  The  OP  model  had  five  category  corroborations,  which  was  the  most  of 
any  model.  Thus,  although  the  results  do  not  permit  a  decisive  determination,  this 
comparative  statistical  study  favors  the  OP  model  as  the  model  of  choice. 
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Table  22.  Corroborated  categories.  In  these  8  categories,  the  same  model  had  the  highest  PE  score 
using  both  energy  flux  thresholds. 


Season 

Kp 

Model 

Winter 

Low 

OP 

Spring 

High 

SWMF 

Spring 

Low 

OP 

Summer 

Low 

OP 

Fall 

Low 

OH 

04-09  MET 

High 

SWMF 

04-09  MET 

Mod 

OP 

16-21  MLT 

Low 

OP 

4.5  Small-scale  Experiment  Using  Predictable  Precipitation  Patterns 

Consideration  of  the  previous  statistical  measures  did  not  confirm  the  existence  of 
a  superior  model.  These  results  underpin  the  difficulties  involved  in  modeling  such  a 
highly  dynamic  phenomenon.  Thus,  the  following  extremely  small-scale  experiment  was 
devised  to  detennine  if  providing  the  models  with  a  small  set  of  well-defined  auroral 
events  would  affect  their  statistical  scores. 

This  study  used  the  data  from  48  satellite  passes.  The  data  was  selected  such  that 
3  events  were  selected  from  16  days.  Each  of  the  48  passes  was  chosen  by  making  a 
qualitative  inspection  of  the  DMSP  data  plots  (see  Figure  16).  An  event  was  selected  if  it 
possessed  a  steep  gradient  at  the  equatorward  boundary,  so  as  to  reduce  the  ambiguity  in 
the  location  of  the  boundary.  Furthermore,  only  data  was  chosen  that  occurred  between 
17-21  MET  to  avoid  the  inherent  difficulties  in  modeling  the  dayside  auroral 
precipitation.  Table  23  shows  how  the  48  individual  events  were  distributed  with  regard 
to  Kp  index. 
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Table  23.  Total  number  of  events  during  specific  Kp  index  value  for  small-scale  study. 


Kp 

3 

4 

5 

6-8 

Number 

13 

13 

14 

8 

Figure  37  depicts  the  data  obtained  from  2  of  the  16  days.  In  the  left  column, 
there  are  three  low-Kp  events  from  the  same  day  ordered  chronologically  from  top  to 
bottom.  The  right  column  shows  three  more  chronological  events  from  a  different  day 
and  during  stormier  conditions.  In  each  plot,  the  dashed  dark  blue  line  represents  the  15- 
second  moving  average  of  the  DMSP  energy  flux  data.  The  NH,  OH,  and  OP  models 
results  are  represented  by  the  solid  green,  cyan,  and  red  lines  respectively. 

Using  0.4  erg/cm2/s  as  the  threshold,  the  DMSP  MLATs  from  the  three 
measurements  taken  during  the  low-Kp  event  occurred  within  a  1 . 1°  span.  The  MLAT 
coordinates  during  the  high-Kp  event  occurred  within  1.5°.  The  three  passes  used  from 
each  day  cover  a  period  of  4-5  hours.  Such  a  well-defined  boundary  location  over  a 
considerable  amount  of  time  represents  relatively  static  auroral  conditions,  reducing  the 
variance  that  pervades  the  DMSP  data  in  larger  samples.  This  stable  boundary  location 
existed  in  the  other  14  days  used  in  this  study  as  well  and  ultimately  provides  a  template 
against  which  the  models  can  be  scrutinized. 

The  sharp  spikes  in  the  red  line  in  the  low  latitudes  demonstrate  the  noisy  nature 
of  the  OP  model  that  was  eliminated  by  imposing  the  stricter  criterion  outlined  in 
Chapter  III.  The  places  where  no  cyan  or  green  line  exists  represent  the  latitudes  at 
which  no  energy  flux  was  calculated  by  either  the  OH  or  NH  model.  The  plots  on  the  left 
show  pictorially  that  the  OH  and  NH  models  depict  the  equatorward  boundary 
consistently  well.  The  intersection  of  the  dark  blue,  yellow,  and  green  lines  with  the 
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black  threshold  line  occurs  within  a  close  proximity.  Furthermore,  the  shape  of  the  NH 
model’s  line  also  depicts  the  steep  gradient  that  closely  approximates  what  is  observed  in 
the  DMSP  data  near  the  boundary.  With  regard  to  the  low-Kp  conditions,  the  OP  model 
tends  to  locate  the  boundary  too  far  equatorward,  with  deviations  approaching  5°  ML  AT. 
And  both  the  OH  and  OP  models  depict  a  more  gradual  energy  flux  onset  than  is  shown 
by  the  DMSP  data,  which  is  inconsistent  with  the  theoretical  understanding  of  the 
nightside  plasma  sheet. 


June  21,  2007  July  10,  2005 


Figure  37.  Chronological  data  plots  of  DMSP  and  model  data  for  low-Kp  (left  column)  and  high-Kp 
(right  column)  events  on  different  days.  Horizontal  black  line  represents  0.4  erg/cm2/s  threshold. 
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The  gradual  onset,  however,  works  in  the  OP  model’s  favor  as  is  shown  in  the 
high-Kp  conditions  depicted  in  the  right  column  of  Figure  37.  Here,  the  DMSP  and  OP 
lines  cross  the  threshold  more  closely  than  was  seen  in  the  low-Kp  event.  However,  it  is 
noted  the  OP  model’s  gradual  flux  onset  (and  to  a  lesser  extent  the  OH  model’s  as  well) 
makes  its  boundary  dependent  upon  the  selected  threshold. 

Having  considered  two  individual  cases,  the  same  statistical  analysis  can  be 
applied  to  the  entire  48-event  dataset.  The  average  deviations  for  each  model  are  shown 
in  Figure  38.  These  results  seen  here  validate  the  intention  of  the  small-scale  study.  The 
nightside  auroral  boundaries  chosen  for  this  micro-study  have  little  spatial  ambiguity 
because  the  energy  flux  onset  occurs  so  rapidly,  and  the  models  have  fewer  (if  any)  large 
deviations. 

Although  this  small  study  has  merely  introduced  the  concept,  the  idea  of 
consistent  and  static  auroral  precipitation  patterns  is  an  obvious  condition  where 
empirically  based  models  like  OH  and  NH  will  excel  because  these  types  of  models  have 
been  assembled  through  statistical  and  regression  analyses  on  a  much  larger  scale. 
Generally  speaking,  the  less  ambiguous  and  more  consistent  auroral  precipitation  regions 
complement  statistically  based  models  like  the  OH  and  NH  models. 

The  scope  of  this  micro  study  is  far  too  small  to  make  any  comprehensive 
categorizations  about  accuracy;  however,  it  has  been  included  to  underscore  the  fact  that 
there  are  aspects  of  the  auroral  oval  that  do  not  exhibit  wild  variations  and  that  are  well- 
suited  to  modeling  by  purely  statistical  means.  In  the  absence  of  this  consistency, 
though,  the  results  obtained  in  the  previous  section  that,  with  refinement,  the  more 
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dynamic,  complex,  and  comprehensive  models  like  OP  do  represent  progress  in  this 
important  regime  of  space  weather  modeling. 


is 


DMSP-OH  MLAT  (degrees) 


DMSP-OP  MLAT  (degrees) 


Figure  38.  Deviation  plots  for  NH,  OH,  and  OP  models. 


Table  24  summarizes  all  of  the  statistics  for  this  small-scale  study.  The  ratio 
estimations  of  the  NH  and  OH  models  show  a  nearly  1 : 1  correspondence  in  boundary 
migration.  It  is  apparent  that  when  considering  a  micro-population  with  a  well-defined 
auroral  boundary,  the  OH  model  is  the  most  successful  model.  It  has  the  highest  PE 
score,  and  the  skill  scores  of  the  NH  and  OP  models  are  negative,  which  also  confinns 
the  OH  model’s  performance  is  superior.  Qualitatively  speaking,  however,  the  closer 
resemblance  between  the  DMSP  satellite’s  energy  profile  and  the  NH  model’s  energy 
flux  output  is  a  favorable  characteristic  of  this  model. 


96 


Table  24.  Prediction  efficiency,  ratio  estimates,  and  skill  scores  for  small-scale  investigation.  The 
OH  has  no  SS  entry  because  it  served  as  the  reference  model. 


NH 

OH 

OP 

PE 

0.30 

0.47 

-0.63 

RE 

0.99 

0.99 

1.04 

SS 

-0.15 

— 

-0.75 

There  is  another  avenue  of  investigation  that  this  small-scale  study  supports. 
Because  of  the  relative  stability  of  the  auroral  oval  over  extended  periods  of  time,  these 
plots  can  also  be  used  to  examine  some  of  the  characteristics  of  the  evolution  of  each  of 
the  models  on  a  relatively  short  time  scale. 

In  Figure  39,  three  chronological  plots  are  shown,  and  each  contains  the  energy 
flux  profile  of  the  DMSP  data  and  the  NH,  OH,  and  OP  models  in  a  specific  MLT  sector. 
For  this  example,  the  time  period  involves  high-Kp  conditions.  The  vertical  lines 
connecting  the  plots  map  depict  the  migration  of  the  threshold  crossing  locations.  This 
particular  example  underscores  some  important  characteristics  of  each  model’s  behavior 
and  highlights  some  positive  and  negative  qualities  that  are  tough  to  discern  from  a 
statistical  study  alone. 

In  the  top  panel  of  the  Figure  39,  the  DMSP  threshold  crossing  is  reasonably  well 
represented  by  all  three  models.  The  NH  model  is  displaced  poleward,  but  as  before,  it’s 
energy  onset  (i.e.,  gradient)  most  closely  mimics  the  DMSP  data.  The  data  in  the  second 
panel  was  obtained  approximately  1  hour  and  40  minutes  later  and  was  accompanied  by  a 
decrease  in  Kp  from  7.0  to  6.0.  Counter  intuitively,  the  DMSP  threshold  crossing 
actually  migrates  slightly  more  toward  the  equator.  There  is  no  change  in  the  NH  or  OH 
model  outputs  because  both  they  are  capped  at  Kp  6.  The  OP  model,  however,  also 
follows  the  DMSP  boundary,  despite  the  fact  that  the  decrease  in  Kp  statistically  warrants 
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poleward  migration.  This  serves  as  a  testament  to  the  benefit  of  a  highly  adaptable  model 
like  OP. 


Figure  39.  Three  chronological  plots  of  the  energy  flux  profiles  of  the  DMSP  data  and  the  NH,  OH, 
and  OP  models.  The  vertical  lines  drawn  between  the  panels  show  the  migration  of  the  threshold 
crossings. 


In  the  third  panel,  again  containing  data  obtained  approximately  1  hour  and  40 
minutes  later,  the  poleward  migration  of  the  threshold  crossing  coordinates  of  the  NH  and 
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OH  models  is  solely  due  to  the  change  in  MLT  sector.  The  Kp  index  remains  the  same 
because  the  data  falls  within  the  three  hour  window  that  was  applicable  to  the  previous 
satellite  pass.  However,  it  is  clear  that  the  auroral  oval  is  relaxing  back  to  a  less  stormy 
state.  Again,  if  the  noise  is  ignored  in  the  region  between  50  and  60°  MLAT,  the  OP 
model  successfully  follows  the  DMSP  data  migration. 

It  is  dangerous  to  make  definitive  conclusions  based  upon  one  example,  but  this 
does  show  the  potential  of  a  using  and  developing  a  highly  adaptable  model  construct  that 
is  not  immune  to  small  changes  in  the  geomagnetic  conditions  that  are  occurring  at  a  far 
faster  rate  than  the  Kp  index  is  updated.  Somewhat  cautiously,  therefore,  it  is  possible  to 
suppose  that  the  future  of  auroral  forecasting  will  continue  to  improve  upon  the  same 
“nowcasting”  capabilities  already  present,  to  some  degree,  in  the  OP  model.  That  being 
said,  the  climatological  utility  of  empirically  derived  models,  like  OH  and  NH,  holds  an 
important  place  in  the  assessment  of  space  weather  conditions. 
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V.  Conclusion 


In  this  chapter,  the  development  of  the  experiment  and  its  results  will  be  briefly 
summarized.  Some  additional  comments  will  be  made  regarding  avenues  of  future 
research. 

5.1  Summary  of  Results 

The  ability  to  accurately  model  the  auroral  oval  is  important  to  national  security 
objectives  in  the  space  domain.  The  auroral  activity  serves  as  a  remarkable  proxy  for  the 
space  weather  environment,  and  the  precipitation  patterns  associated  with  it  have  a  direct 
impact  on  space-based  operations,  especially  including  communication  and  navigation 
capabilities. 

This  study  obtained  data  from  more  than  6,000  DMSP  satellites  passes  between 
2000-2008.  This  data  was  carefully  quality  checked  and  vetted  against  previous  studies 
of  a  similar  nature.  It  was  detennined  that,  despite  a  high  degree  of  variance,  the 
characteristics  and  location  of  the  equatorward  boundary  could  be  effectively  determined 
using  the  DMSP  data. 

The  five  models  utilized  in  this  study  brought  to  bear  several  different  approaches 
to  modeling,  and  the  characteristic  behavior  of  each  of  the  models  under  a  variety  of 
conditions  was  investigated  in  detail.  This  exposed  the  nature  of  the  data  loss  problem 
that  was  encountered  when  the  energy  flux  threshold  was  set  too  high.  It  also  revealed  a 
systematic  noise  issue  with  the  low-latitude  output  of  the  OP  model,  and  it  underscored 
the  limitations  of  models  like  SWMF  and  AMIE,  whose  resolutions  were  significantly 
coarser  than  the  other  models. 
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Each  model’s  output  was  compared  to  DMSP  data,  and  several  basic  statistics 
were  obtained.  Calculations  of  each  model’s  mean  deviations  and  the  variance  of  these 
deviations  led  to  the  validation  of  the  NH,  OH,  OP,  and  SWMF  as  viable  models  under 
certain  conditions.  Then,  the  prediction  efficiency  of  each  model  was  calculated  using 
the  0.4  erg/cm2/s  threshold  dataset  because  it  was  determined  to  have  the  highest  validity. 

Calculating  the  prediction  efficiency  scores  led  to  the  conclusion  that  the  OP 
model  was  the  most  accurate,  but  each  model  had  comparative  success  when  certain 
criteria  were  imposed.  The  OH  model  received  high  scores  for  data  obtained  between 
10-15  MLT.  The  OP  model’s  PE  scores  were  high  during  low-Kp  conditions.  The 
SWMF  model  was  extremely  effective  during  high-Kp  conditions,  regardless  of  season  or 
MLT  sector.  Here,  it  was  also  ascertained  that  the  AMIE  model  underperformed  in  all 
categories. 

The  results  found  using  a  0.4  erg/cnr/s  threshold  were  corroborated  using  a 
dataset  obtained  with  a  0.6  erg/cnr/s  threshold.  The  PE  scores  at  this  threshold  showed  a 
substantial  uptick  in  the  performance  of  the  OP  model,  which  was  ultimately  attributed  to 
a  lack  of  adaptation  of  the  NH  and  OH  models  to  the  new  threshold.  Of  the  18  categories 
in  which  data  was  available,  the  same  model  had  the  highest  PE  score  8  times,  and  the 
OP  model  was  responsible  for  5  of  those  8  occurrences.  From  these  results,  the  OP 
model  was  determined  to  be  the  most  unifonnly  accurate  model. 

Skill  scores  were  also  used  to  measure  the  perfonnance  of  the  NH  and  OP  models 
using  the  OH  model  as  the  reference.  At  the  low  energy  threshold,  both  models  generally 
performed  better  than  OH.  Ultimately,  the  skill  score  analysis  did  not  contribute  much  to 
the  distinction  between  the  model’s  performances. 
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Finally,  a  small-scale  study  was  conducted  utilizing  48  hand-selected  examples  of 
nightside  aurora  with  well-defined  equatorward  boundaries.  These  cases  were  selected  in 
order  to  study  and  categorize  the  models’  performance  during  relatively  static  auroral 
precipitation  conditions.  The  results  of  this  micro  study  led  to  the  conclusion  that  the  OH 
model  was  the  most  accurate  at  modeling  conditions  with  an  unambiguous  boundary, 
which  is  consistent  with  its  statistically  based  development.  However,  it  was  also  shown 
that  the  adaptable  design  of  the  OP  model  was  a  modeling  capability  that  distinctly 
exceeded  both  of  the  highly  discretized  Hardy  models. 

Based  upon  these  factors,  the  OVATION  Prime  model  has  manifested  itself  as  the 
most  suitable  candidate  for  operational  employment.  Foremost,  it  exhibited  the  highest 
degree  of  accuracy  when  using  a  fixed  energy  flux  threshold  to  depict  the  geographic 
extent  of  the  auroral  oval,  and  this  accuracy  was  corroborated  at  multiple  thresholds.  Its 
equatorward  bias  detracts  from  its  accuracy,  but  this  behavior  is  operationally  more 
conservative  than  is  underestimating  the  oval’s  extent.  The  OP  model  also  offers  a 
highly  adaptable  approach  to  modeling,  which  presents  an  important  capability, 
especially  during  high-Kp  conditions. 

5.2  Future  Research 

The  scope  of  this  research  was  truly  broad,  and  opportunities  for  future  research 
exist  along  many  different  avenues.  At  the  outset,  it  would  certainly  be  warranted  to 
recreate  the  statistics  obtained  in  this  study  with  a  larger  data  set  by  utilizing  more  of  the 
available  DMSP  data.  There  exists,  however,  an  inherent  challenge  in  utilizing  DMSP 
data,  and  it  is  important  some  sort  of  quality  standard  is  enforced.  This  is  the  most  time 
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consuming  element  of  a  study  of  this  sort,  but  an  enhanced  computer  algorithm  to 
streamline  the  process  is  within  the  realm  of  possibility. 

An  additional  avenue  of  complementary  research  pertains  to  the  criterion/criteria 
by  which  the  equatorward  boundary  is  defined.  This  study  took  a  somewhat  simplistic 
approach  by  looking  at  a  fixed  energy  flux  threshold.  Exploring  other  means  of 
identifying  the  boundary  would  certainly  be  a  fair  and  useful  way  to  corroborate  the 
results  found  in  this  study. 

Finally,  the  use  of  a  different  source  of  data  rather  than  single  pass  particle  counts 
from  the  DMSP  satellites  may  yield  fruitful  results.  There  are  a  multitude  of  additional 
sources  that  can  be  used  to  directly  measure  or  infer  the  equatorward  boundary  of  the 
auroral  oval.  Synoptic  measurements,  such  as  those  obtained  with  all-sky  cameras  or  the 
Polar  satellites  UVI  instrument,  could  prove  to  be  excellent  tools  for  capturing  this  highly 
dynamic  phenomenon. 
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